2025-06-02 Mon 02:08:24:236.802 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: vhost interface name : vhost0 2025-06-02 Mon 02:08:24:236.972 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: vhost IP Address : 10.0.0.129/24 2025-06-02 Mon 02:08:24:236.980 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: vhost gateway : 0.0.0.0 2025-06-02 Mon 02:08:24:236.986 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Ethernet ports : ens3 2025-06-02 Mon 02:08:24:236.991 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Loopback IP : 0.0.0.0 2025-06-02 Mon 02:08:24:236.996 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Servers : 10.0.0.129:5269 2025-06-02 Mon 02:08:24:237.002 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Authentication : 1 2025-06-02 Mon 02:08:24:237.007 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Server Certificate : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:24:237.012 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Server Key : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:24:237.016 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp CA Certificate : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:24:237.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Cluster-Name : 2025-06-02 Mon 02:08:24:237.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: DNS Servers : 10.0.0.129:53 2025-06-02 Mon 02:08:24:237.031 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: DNS client port : 8098 2025-06-02 Mon 02:08:24:237.035 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: DNS timeout : 3000 2025-06-02 Mon 02:08:24:237.039 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: DNS max retries : 2 2025-06-02 Mon 02:08:24:237.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Dns Authentication : 1 2025-06-02 Mon 02:08:24:237.051 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Server Certificate : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:24:237.057 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp Server Key : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:24:237.063 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Xmpp CA Certificate : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:24:237.069 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: TSN Servers : [] 2025-06-02 Mon 02:08:24:237.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: COLLECTOR Servers : 10.0.0.129:8086 2025-06-02 Mon 02:08:24:237.081 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Tunnel-Type : MPLSoGRE 2025-06-02 Mon 02:08:24:237.102 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Metadata-Proxy Shared Secret: 8bbe7903-8b3e-4e94-9237-98c3349d0bce 2025-06-02 Mon 02:08:24:237.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Metadata-Proxy Port : 8097 2025-06-02 Mon 02:08:24:237.117 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Metadata-Proxy SSL Flag : 0 2025-06-02 Mon 02:08:24:237.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Max Vm Flows : 100 2025-06-02 Mon 02:08:24:237.149 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Linklocal Max System Flows : 16384 2025-06-02 Mon 02:08:24:237.154 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Linklocal Max Vm Flows : 16384 2025-06-02 Mon 02:08:24:237.158 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow cache timeout : 0 2025-06-02 Mon 02:08:24:237.162 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Stale Interface cleanup timeout : 60 2025-06-02 Mon 02:08:24:237.167 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow thread count : 1 2025-06-02 Mon 02:08:24:237.173 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow latency limit : 0 2025-06-02 Mon 02:08:24:237.179 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow index-mgr sm log count : 0 2025-06-02 Mon 02:08:24:237.184 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow add-tokens : 50 2025-06-02 Mon 02:08:24:237.191 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow ksync-tokens : 25 2025-06-02 Mon 02:08:24:237.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow del-tokens : 16 2025-06-02 Mon 02:08:24:237.203 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow update-tokens : 16 2025-06-02 Mon 02:08:24:237.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Pin flow netlink task to CPU: 2025-06-02 Mon 02:08:24:237.217 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Maximum sessions : 100 2025-06-02 Mon 02:08:24:237.222 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Maximum session aggregates : 8 2025-06-02 Mon 02:08:24:237.226 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Maximum session endpoints : 5 2025-06-02 Mon 02:08:24:237.233 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Fabric SNAT hash table size : 4096 2025-06-02 Mon 02:08:24:237.238 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Flow excluding Router ID in hash :0 2025-06-02 Mon 02:08:24:237.242 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Agent Mode : Vrouter 2025-06-02 Mon 02:08:24:237.247 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Gateway Mode : None 2025-06-02 Mon 02:08:24:237.251 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: DHCP Relay Mode : 0 2025-06-02 Mon 02:08:24:237.255 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance netns cmd : /usr/local/bin/opencontrail-vrouter-netns 2025-06-02 Mon 02:08:24:237.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance docker cmd : /usr/local/bin/opencontrail-vrouter-docker 2025-06-02 Mon 02:08:24:237.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance workers : 0 2025-06-02 Mon 02:08:24:237.268 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance timeout : 0 2025-06-02 Mon 02:08:24:237.272 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance lb ssl : 2025-06-02 Mon 02:08:24:237.276 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Service instance lbaas auth : 2025-06-02 Mon 02:08:24:237.281 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Bgp as a service port range : 50000-50512 2025-06-02 Mon 02:08:24:237.291 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Services queue limit : 1024 2025-06-02 Mon 02:08:24:237.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: BGPAAS max shared sessions for service port : 4 2025-06-02 Mon 02:08:24:237.300 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Sandesh Key file : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:24:237.304 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Sandesh Cert file : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:24:237.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Sandesh CA Cert : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:24:237.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Sandesh SSL Enable : 1 2025-06-02 Mon 02:08:24:237.316 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Introspect SSL Enable : 1 2025-06-02 Mon 02:08:24:237.320 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Hypervisor mode : kvm 2025-06-02 Mon 02:08:24:237.427 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Ethernet Port Encap Type : 2025-06-02 Mon 02:08:24:237.434 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: Platform mode : Vrouter on host linux kernel 2025-06-02 Mon 02:08:24:237.786 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: VRouter [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"nightly","build-time":"2025-06-02 00:26:11.811431","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-9311-1.novalocal","build-id":"unknown","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-06-02 Mon 02:08:24:237.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Logging: DISABLED -> ENABLED 2025-06-02 Mon 02:08:24:237.862 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-06-02 Mon 02:08:24:266.132 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.169 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> self 2025-06-02 Mon 02:08:24:266.175 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:24:266.181 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:24:266.186 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.191 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-access-control-list -> self 2025-06-02 Mon 02:08:24:266.196 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap access-control-list : self -> virtual-network-access-control-list 2025-06-02 Mon 02:08:24:266.201 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap access-control-list : virtual-network-access-control-list -> nil 2025-06-02 Mon 02:08:24:266.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.217 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> self 2025-06-02 Mon 02:08:24:266.222 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:24:266.227 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> virtual-network-network-ipam 2025-06-02 Mon 02:08:24:266.232 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:24:266.237 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.242 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-qos-config -> self 2025-06-02 Mon 02:08:24:266.247 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-config : self -> virtual-network-qos-config 2025-06-02 Mon 02:08:24:266.252 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-config : virtual-network-qos-config -> nil 2025-06-02 Mon 02:08:24:266.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.278 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-security-logging-object -> self 2025-06-02 Mon 02:08:24:266.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : self -> virtual-network-security-logging-object 2025-06-02 Mon 02:08:24:266.288 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : virtual-network-security-logging-object -> nil 2025-06-02 Mon 02:08:24:266.293 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : logical-router-virtual-network -> self 2025-06-02 Mon 02:08:24:266.302 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router-virtual-network : self -> logical-router-virtual-network 2025-06-02 Mon 02:08:24:266.307 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router-virtual-network : logical-router-virtual-network -> logical-router-virtual-network 2025-06-02 Mon 02:08:24:266.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router : self -> logical-router-virtual-network 2025-06-02 Mon 02:08:24:266.317 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router : logical-router-virtual-network -> nil 2025-06-02 Mon 02:08:24:266.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:24:266.326 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-multicast-policy -> self 2025-06-02 Mon 02:08:24:266.331 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap multicast-policy : self -> virtual-network-multicast-policy 2025-06-02 Mon 02:08:24:266.338 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap multicast-policy : virtual-network-multicast-policy -> nil 2025-06-02 Mon 02:08:24:266.344 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for routing-instance 2025-06-02 Mon 02:08:24:266.348 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> self 2025-06-02 Mon 02:08:24:266.353 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:24:266.358 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-routing-instance 2025-06-02 Mon 02:08:24:266.362 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:24:266.367 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:24:266.373 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for security-group 2025-06-02 Mon 02:08:24:266.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-group : security-group-access-control-list -> self 2025-06-02 Mon 02:08:24:266.382 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap access-control-list : self -> security-group-access-control-list 2025-06-02 Mon 02:08:24:266.386 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap access-control-list : security-group-access-control-list -> nil 2025-06-02 Mon 02:08:24:266.392 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for tag 2025-06-02 Mon 02:08:24:266.397 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> self 2025-06-02 Mon 02:08:24:266.402 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:266.406 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : policy-management-application-policy-set -> application-policy-set-tag 2025-06-02 Mon 02:08:24:266.411 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap policy-management : policy-management-application-policy-set -> nil 2025-06-02 Mon 02:08:24:266.416 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.421 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:24:266.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:266.430 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:266.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:24:266.442 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:24:266.447 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.452 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-06-02 Mon 02:08:24:266.456 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:24:266.460 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : virtual-machine-interface-virtual-machine -> nil 2025-06-02 Mon 02:08:24:266.465 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.470 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-06-02 Mon 02:08:24:266.474 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:24:266.479 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> nil 2025-06-02 Mon 02:08:24:266.483 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.488 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:24:266.492 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:266.497 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:266.501 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:24:266.506 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:24:266.511 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.515 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-routing-instance -> self 2025-06-02 Mon 02:08:24:266.520 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-routing-instance : self -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:24:266.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-routing-instance : virtual-machine-interface-routing-instance -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:24:266.530 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : self -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:24:266.537 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : virtual-machine-interface-routing-instance -> nil 2025-06-02 Mon 02:08:24:266.542 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.546 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-group -> self 2025-06-02 Mon 02:08:24:266.551 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-group : self -> virtual-machine-interface-security-group 2025-06-02 Mon 02:08:24:266.555 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-group : virtual-machine-interface-security-group -> nil 2025-06-02 Mon 02:08:24:266.561 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.565 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : alias-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.570 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap alias-ip : self -> alias-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.575 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap alias-ip : alias-ip-pool-alias-ip -> alias-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.580 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap alias-ip-pool : virtual-network-alias-ip-pool -> alias-ip-pool-alias-ip 2025-06-02 Mon 02:08:24:266.585 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-alias-ip-pool 2025-06-02 Mon 02:08:24:266.590 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-alias-ip-pool -> nil 2025-06-02 Mon 02:08:24:266.617 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.627 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.631 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.637 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-06-02 Mon 02:08:24:266.641 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:24:266.646 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:24:266.650 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:24:266.659 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:24:266.664 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.669 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.674 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.678 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.682 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-06-02 Mon 02:08:24:266.687 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:24:266.691 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:24:266.696 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:24:266.700 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:24:266.706 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.710 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.714 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.719 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap floating-ip : instance-ip-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.724 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap instance-ip : instance-ip-virtual-network -> instance-ip-floating-ip 2025-06-02 Mon 02:08:24:266.728 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> instance-ip-virtual-network 2025-06-02 Mon 02:08:24:266.733 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : instance-ip-virtual-network -> nil 2025-06-02 Mon 02:08:24:266.738 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.742 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : instance-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.746 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap instance-ip : self -> instance-ip-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.751 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap instance-ip : instance-ip-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:24:266.759 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.767 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-route-table -> self 2025-06-02 Mon 02:08:24:266.772 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap interface-route-table : self -> virtual-machine-interface-route-table 2025-06-02 Mon 02:08:24:266.777 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap interface-route-table : virtual-machine-interface-route-table -> nil 2025-06-02 Mon 02:08:24:266.782 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.786 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : subnet-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.792 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap subnet : self -> subnet-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.796 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap subnet : subnet-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:24:266.802 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.806 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> logical-interface-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.957 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-06-02 Mon 02:08:24:266.963 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:24:266.967 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:24:266.974 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:266.978 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-port-group-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:266.984 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> virtual-port-group-virtual-machine-interface 2025-06-02 Mon 02:08:24:266.988 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : physical-router-physical-interface -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:24:266.993 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:24:266.998 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:24:267.007 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.011 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:267.016 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-as-a-service : bgpaas-bgp-router -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-router : self -> bgpaas-bgp-router 2025-06-02 Mon 02:08:24:267.031 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-router : instance-bgp-router -> bgpaas-bgp-router 2025-06-02 Mon 02:08:24:267.036 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : self -> instance-bgp-router 2025-06-02 Mon 02:08:24:267.040 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : instance-bgp-router -> nil 2025-06-02 Mon 02:08:24:267.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.049 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:267.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.058 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-as-a-service : bgpaas-health-check -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-health-check : self -> bgpaas-health-check 2025-06-02 Mon 02:08:24:267.068 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-health-check : bgpaas-health-check -> nil 2025-06-02 Mon 02:08:24:267.073 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.078 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-qos-config -> self 2025-06-02 Mon 02:08:24:267.082 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-config : self -> virtual-machine-interface-qos-config 2025-06-02 Mon 02:08:24:267.087 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-config : virtual-machine-interface-qos-config -> nil 2025-06-02 Mon 02:08:24:267.092 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-logging-object -> self 2025-06-02 Mon 02:08:24:267.101 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : self -> virtual-machine-interface-security-logging-object 2025-06-02 Mon 02:08:24:267.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : virtual-machine-interface-security-logging-object -> nil 2025-06-02 Mon 02:08:24:267.114 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.118 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-06-02 Mon 02:08:24:267.123 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.143 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:24:267.147 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-06-02 Mon 02:08:24:267.152 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.157 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-06-02 Mon 02:08:24:267.161 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.165 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.170 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:24:267.175 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : virtual-machine-interface-bridge-domain -> nil 2025-06-02 Mon 02:08:24:267.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.201 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-tag -> self 2025-06-02 Mon 02:08:24:267.206 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> virtual-machine-interface-tag 2025-06-02 Mon 02:08:24:267.211 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-interface-tag 2025-06-02 Mon 02:08:24:267.215 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.254 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.260 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.270 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.274 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:267.281 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.285 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:24:267.290 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:267.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-tag -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:267.299 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> virtual-network-tag 2025-06-02 Mon 02:08:24:267.304 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> virtual-network-tag 2025-06-02 Mon 02:08:24:267.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.313 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.317 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.326 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.330 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:267.336 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.340 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-06-02 Mon 02:08:24:267.347 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:24:267.352 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:24:267.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-06-02 Mon 02:08:24:267.361 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-06-02 Mon 02:08:24:267.365 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.370 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.374 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.378 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.383 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.387 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:267.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.398 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-06-02 Mon 02:08:24:267.402 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:24:267.407 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:24:267.411 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:24:267.416 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:24:267.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-06-02 Mon 02:08:24:267.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-06-02 Mon 02:08:24:267.429 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.433 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.444 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.449 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.453 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:267.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : project-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:267.473 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap project : self -> project-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.477 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap project : project-tag -> project-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> project-tag 2025-06-02 Mon 02:08:24:267.486 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : application-policy-set-tag -> project-tag 2025-06-02 Mon 02:08:24:267.491 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.495 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:24:267.500 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.508 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:267.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:267.518 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.522 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> self 2025-06-02 Mon 02:08:24:267.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.532 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.537 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.544 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> nil 2025-06-02 Mon 02:08:24:267.549 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:24:267.554 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : logical-router-interface -> self 2025-06-02 Mon 02:08:24:267.558 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router : self -> logical-router-interface 2025-06-02 Mon 02:08:24:267.563 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-router : logical-router-interface -> nil 2025-06-02 Mon 02:08:24:267.569 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for physical-interface 2025-06-02 Mon 02:08:24:267.573 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : physical-router-physical-interface -> self 2025-06-02 Mon 02:08:24:267.578 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:24:267.582 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:24:267.587 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for physical-interface 2025-06-02 Mon 02:08:24:267.602 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : virtual-port-group-physical-interface -> self 2025-06-02 Mon 02:08:24:267.607 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-port-group-physical-interface : self -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:24:267.612 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-port-group-physical-interface : virtual-port-group-physical-interface -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:24:267.616 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-port-group : self -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:24:267.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> nil 2025-06-02 Mon 02:08:24:267.626 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:24:267.630 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> self 2025-06-02 Mon 02:08:24:267.635 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : self -> physical-interface-logical-interface 2025-06-02 Mon 02:08:24:267.639 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-06-02 Mon 02:08:24:267.644 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:24:267.651 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:24:267.656 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:24:267.660 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-interface : logical-interface-virtual-machine-interface -> self 2025-06-02 Mon 02:08:24:267.665 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> logical-interface-virtual-machine-interface 2025-06-02 Mon 02:08:24:267.669 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:24:267.674 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:24:267.679 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap logical-interface : physical-router-logical-interface -> self 2025-06-02 Mon 02:08:24:267.683 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : self -> physical-router-logical-interface 2025-06-02 Mon 02:08:24:267.688 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-logical-interface -> nil 2025-06-02 Mon 02:08:24:267.693 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for physical-router 2025-06-02 Mon 02:08:24:267.697 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-router : physical-router-physical-interface -> self 2025-06-02 Mon 02:08:24:267.702 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:24:267.706 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap physical-interface : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:24:267.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for service-health-check 2025-06-02 Mon 02:08:24:267.717 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-06-02 Mon 02:08:24:267.721 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-06-02 Mon 02:08:24:267.726 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> service-port-health-check 2025-06-02 Mon 02:08:24:267.731 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:267.735 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:24:267.740 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:24:267.746 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for service-health-check 2025-06-02 Mon 02:08:24:267.753 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-06-02 Mon 02:08:24:267.757 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-06-02 Mon 02:08:24:267.762 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> service-port-health-check 2025-06-02 Mon 02:08:24:267.766 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.770 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.775 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.779 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> port-tuple-interface 2025-06-02 Mon 02:08:24:267.784 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:24:267.788 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:24:267.793 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:24:267.798 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for qos-config 2025-06-02 Mon 02:08:24:267.802 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-config : global-qos-config-qos-config -> self 2025-06-02 Mon 02:08:24:267.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap global-qos-config : global-qos-config-qos-config -> nil 2025-06-02 Mon 02:08:24:267.814 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for forwarding-class 2025-06-02 Mon 02:08:24:267.819 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap forwarding-class : forwarding-class-qos-queue -> self 2025-06-02 Mon 02:08:24:267.824 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-queue : self -> forwarding-class-qos-queue 2025-06-02 Mon 02:08:24:267.828 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap qos-queue : forwarding-class-qos-queue -> nil 2025-06-02 Mon 02:08:24:267.834 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:24:267.838 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : security-logging-object-network-policy -> self 2025-06-02 Mon 02:08:24:267.849 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap network-policy : self -> security-logging-object-network-policy 2025-06-02 Mon 02:08:24:267.853 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap network-policy : security-logging-object-network-policy -> nil 2025-06-02 Mon 02:08:24:267.861 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:24:267.865 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : security-logging-object-security-group -> self 2025-06-02 Mon 02:08:24:267.870 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-group : self -> security-logging-object-security-group 2025-06-02 Mon 02:08:24:267.874 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-group : security-logging-object-security-group -> nil 2025-06-02 Mon 02:08:24:267.879 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:24:267.884 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : firewall-policy-security-logging-object -> self 2025-06-02 Mon 02:08:24:267.889 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-security-logging-object : self -> firewall-policy-security-logging-object 2025-06-02 Mon 02:08:24:267.894 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-security-logging-object : firewall-policy-security-logging-object -> firewall-policy-security-logging-object 2025-06-02 Mon 02:08:24:267.899 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : firewall-policy-security-logging-object -> nil 2025-06-02 Mon 02:08:24:267.904 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:24:267.908 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap security-logging-object : firewall-rule-security-logging-object -> self 2025-06-02 Mon 02:08:24:267.913 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule-security-logging-object : self -> firewall-rule-security-logging-object 2025-06-02 Mon 02:08:24:267.918 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule-security-logging-object : firewall-rule-security-logging-object -> firewall-rule-security-logging-object 2025-06-02 Mon 02:08:24:267.923 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : firewall-rule-security-logging-object -> nil 2025-06-02 Mon 02:08:24:267.933 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for virtual-router 2025-06-02 Mon 02:08:24:267.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-router : virtual-router-sub-cluster -> self 2025-06-02 Mon 02:08:24:267.944 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap sub-cluster : self -> virtual-router-sub-cluster 2025-06-02 Mon 02:08:24:267.948 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap sub-cluster : bgp-router-sub-cluster -> virtual-router-sub-cluster 2025-06-02 Mon 02:08:24:267.953 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-router : self -> bgp-router-sub-cluster 2025-06-02 Mon 02:08:24:267.957 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bgp-router : bgp-router-sub-cluster -> nil 2025-06-02 Mon 02:08:24:267.966 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for bridge-domain 2025-06-02 Mon 02:08:24:267.976 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-06-02 Mon 02:08:24:267.983 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:24:267.990 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:24:267.997 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:24:268.004 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:24:268.011 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for bridge-domain 2025-06-02 Mon 02:08:24:268.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-06-02 Mon 02:08:24:268.022 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:24:268.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-06-02 Mon 02:08:24:268.032 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for application-policy-set 2025-06-02 Mon 02:08:24:268.037 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> self 2025-06-02 Mon 02:08:24:268.041 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:268.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:268.050 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:24:268.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:24:268.060 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:24:268.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:24:268.069 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.078 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : firewall-rule-service-group -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.091 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-group : self -> firewall-rule-service-group 2025-06-02 Mon 02:08:24:268.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap service-group : firewall-rule-service-group -> nil 2025-06-02 Mon 02:08:24:268.101 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:24:268.106 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:24:268.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.114 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.123 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : firewall-rule-address-group -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap address-group : self -> firewall-rule-address-group 2025-06-02 Mon 02:08:24:268.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap address-group : address-group-tag -> firewall-rule-address-group 2025-06-02 Mon 02:08:24:268.137 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> address-group-tag 2025-06-02 Mon 02:08:24:268.142 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : address-group-tag -> nil 2025-06-02 Mon 02:08:24:268.147 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:24:268.151 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:24:268.155 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.164 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.168 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : firewall-rule-tag -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.173 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : self -> firewall-rule-tag 2025-06-02 Mon 02:08:24:268.177 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap tag : firewall-rule-tag -> nil 2025-06-02 Mon 02:08:24:268.182 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:24:268.189 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:24:268.193 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.198 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.202 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:24:268.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Adding ReactorMap firewall-rule : firewall-policy-firewall-rule -> nil 2025-06-02 Mon 02:08:24:441.539 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.0.0.129:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:24:443.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Current receive sock buffer size is 262144 2025-06-02 Mon 02:08:24:443.188 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013167818496, Pid 1165409]: KsyncTxQueue CPU pinning policy <>. KsyncTxQueuen not pinned to CPU 2025-06-02 Mon 02:08:24:444.428 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Vrouter family is 34 2025-06-02 Mon 02:08:24:444.681 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Flow table size : 20185088 2025-06-02 Mon 02:08:24:444.752 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Flow table size : 161218560 2025-06-02 Mon 02:08:24:444.767 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SetHugePages: Bridge table size:20185088 Flow table size:161218560 2025-06-02 Mon 02:08:24:444.870 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Mem mapping hugepage file:/dev/hugepages2M/bridge size:20185088 2025-06-02 Mon 02:08:24:445.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Mem mapped hugepage file:/dev/hugepages2M/bridge to addr:0x7f5757e00000 2025-06-02 Mon 02:08:24:445.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Mem mapping hugepage file:/dev/hugepages2M/flow size:161218560 2025-06-02 Mon 02:08:24:445.948 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Mem mapped hugepage file:/dev/hugepages2M/flow to addr:0x7f574e400000 2025-06-02 Mon 02:08:24:445.964 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Sending Huge Page configuration to VROUTER 2025-06-02 Mon 02:08:24:501.760 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Huge pages set successfully 2025-06-02 Mon 02:08:24:516.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Flow table size : 161218560 2025-06-02 Mon 02:08:24:516.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Flow table size : 20185088 2025-06-02 Mon 02:08:24:597.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: ROLE : Generator 2025-06-02 Mon 02:08:24:597.115 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: MODULE : contrail-vrouter-agent 2025-06-02 Mon 02:08:24:597.123 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: SOURCE : cn-jenkins-deploy-platform-juju-os-542-1 2025-06-02 Mon 02:08:24:597.137 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: NODE TYPE : Compute 2025-06-02 Mon 02:08:24:597.141 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: INSTANCE ID : 0 2025-06-02 Mon 02:08:24:597.146 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: HTTP SERVER PORT : 8085 2025-06-02 Mon 02:08:24:597.152 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-06-02 Mon 02:08:24:598.456 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: HTTP Introspect Init 2025-06-02 Mon 02:08:24:598.835 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8085 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:24:598.933 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Sandesh Http Server Port: 8085 2025-06-02 Mon 02:08:24:599.013 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: NOT Writing http_port 8085TO : /tmp/contrail-vrouter-agent.1164607.http_port 2025-06-02 Mon 02:08:24:599.024 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: CONNECT TO COLLECTOR: 1 2025-06-02 Mon 02:08:24:599.156 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Idle 2025-06-02 Mon 02:08:24:600.265 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: No Client: 1748830104599281 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:24:601.177 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvStart in state Idle 2025-06-02 Mon 02:08:24:601.906 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Disconnect 2025-06-02 Mon 02:08:24:602.067 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvCollectorUpdate in state Disconnect 2025-06-02 Mon 02:08:24:602.336 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:24:602.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:602.450 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104601890 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:24:602.461 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:24:602.471 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.315 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602018 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:__default__ index = 0 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-06-02 Mon 02:08:24:604.341 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-06-02 Mon 02:08:24:604.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.446 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602054 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:24:604.468 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:24:604.481 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.496 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602069 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:ip-fabric index = 1 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-06-02 Mon 02:08:24:604.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-06-02 Mon 02:08:24:604.513 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.638 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602108 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:24:604.656 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:24:604.670 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.699 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602146 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:24:604.707 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:24:604.718 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:604.838 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104602406 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:24:604.852 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:24:605.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface lo flags 65609 2025-06-02 Mon 02:08:24:605.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface ens3 flags 69699 2025-06-02 Mon 02:08:24:605.166 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface fan-252 flags 69699 2025-06-02 Mon 02:08:24:605.176 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface ftun0 flags 69699 2025-06-02 Mon 02:08:24:605.218 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface lxdbr0 flags 4099 2025-06-02 Mon 02:08:24:605.233 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vethe884fb7d flags 69699 2025-06-02 Mon 02:08:24:605.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vethd808c2e3 flags 69699 2025-06-02 Mon 02:08:24:605.304 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth23343b0c flags 69699 2025-06-02 Mon 02:08:24:605.368 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vethcc1cc497 flags 69699 2025-06-02 Mon 02:08:24:605.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth9204b2f6 flags 69699 2025-06-02 Mon 02:08:24:605.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth5b2dff57 flags 69699 2025-06-02 Mon 02:08:24:605.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vethcc87f2d4 flags 69699 2025-06-02 Mon 02:08:24:605.480 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth515b420e flags 69699 2025-06-02 Mon 02:08:24:605.489 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth90da5e1b flags 69699 2025-06-02 Mon 02:08:24:605.713 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth1d674a27 flags 69699 2025-06-02 Mon 02:08:24:605.735 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface veth2c304bf7 flags 69699 2025-06-02 Mon 02:08:24:605.778 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vethf9f75916 flags 69699 2025-06-02 Mon 02:08:24:605.785 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface docker0 flags 4099 2025-06-02 Mon 02:08:24:605.812 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface pkt1 flags 65601 2025-06-02 Mon 02:08:24:605.819 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface pkt3 flags 65601 2025-06-02 Mon 02:08:24:605.826 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface pkt2 flags 65601 2025-06-02 Mon 02:08:24:605.836 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface vhost0 flags 69699 2025-06-02 Mon 02:08:24:605.842 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink interface message RTM_NEWLINK for interface pkt0 flags 69699 2025-06-02 Mon 02:08:24:605.917 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface lo 2025-06-02 Mon 02:08:24:605.932 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-06-02 Mon 02:08:24:605.944 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface lxdbr0 2025-06-02 Mon 02:08:24:605.956 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface docker0 2025-06-02 Mon 02:08:24:605.968 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-06-02 Mon 02:08:24:605.994 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface lo 2025-06-02 Mon 02:08:24:606.006 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-06-02 Mon 02:08:24:606.018 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface ftun0 2025-06-02 Mon 02:08:24:606.030 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-06-02 Mon 02:08:24:606.041 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink address message RTM_NEWADDR for interface pkt0 2025-06-02 Mon 02:08:24:606.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.143 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.163 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.0.0.1/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.177 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.0.0.2/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.190 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.0.0.4/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.256 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 10.91.248.0/24 Interface lxdbr0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.278 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 172.17.0.0/16 Interface docker0 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.292 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Handle netlink route message RTM_NEWROUTE : 252.0.0.0/8 Interface fan-252 GW 0.0.0.0 2025-06-02 Mon 02:08:24:606.300 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.306 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:24:606.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.318 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:24:606.324 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.330 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.336 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:24:606.366 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.372 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:24:606.378 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:24:606.384 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:24:606.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.445 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:24:606.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:24:606.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-06-02 Mon 02:08:24:606.478 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-06-02 Mon 02:08:24:606.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:24:606.486 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:24:606.490 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:24:606.494 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:24:606.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:24:606.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:24:606.507 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:24:632.843 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:9091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:24:633.060 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Router ID configured. Connection to controller initiated 2025-06-02 Mon 02:08:24:640.073 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:640.124 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104640002 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:640.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:640.926 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:640.976 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104640090 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.129, Collector connection down, No Configuration for self Number of connections:2, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:24:640.995 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:24:641.005 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:641.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104640409 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:641.023 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:641.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:641.049 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104640430 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.129, XMPP:dns-server:10.0.0.129, Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:24:641.057 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:24:642.087 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: TCP [SYS_ERR]: TcpServerMessageLog: Server TCP bind(fe80::f816:3eff:fefe:bee4%vhost0:8097): Cannot assign requested address src/contrail-common/io/tcp_server.cc 111 2025-06-02 Mon 02:08:24:685.737 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:685.782 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: SANDESH: Send FAILED: 1748830104685703 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:685.791 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:685.900 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:685.941 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104685873 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect event = Connect Event local_port = 37953 remote_port = 5269 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:685.950 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:694.381 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:37953::10.0.0.129:5269 < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-06-02 Mon 02:08:24:694.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Event: Tcp Connected peer ip: 10.0.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1295 2025-06-02 Mon 02:08:24:695.828 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:695.869 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104695802 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect next_state = OpenSent event = EvTcpConnected local_port = 37953 remote_port = 5269 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:695.878 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:695.886 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:695.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104695828 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:695.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:696.008 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:696.018 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104696001 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect event = Connect Event local_port = 37735 remote_port = 8093 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:696.024 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:698.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:37735::10.0.0.129:8093 < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-06-02 Mon 02:08:24:698.141 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Event: Tcp Connected peer ip: 10.0.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1295 2025-06-02 Mon 02:08:24:698.268 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:698.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104698248 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect next_state = OpenSent event = EvTcpConnected local_port = 37735 remote_port = 8093 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:698.319 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:698.385 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:698.423 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830104698371 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenSent next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:698.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:698.443 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:698.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830104698405 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Sent Start Tls, OpenConfirm Feature Negotiation event = EvStreamFeatureRequest ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:698.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:698.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:24:698.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013146826496, Pid 1165409]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:24:698.792 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013138429696, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:698.837 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013138429696, Pid 1165409]: SANDESH: Send FAILED: 1748830104698772 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Trigger Client Ssl Handshake event = EvTlsProceed identifier = network-control@contrailsystems.com ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:698.855 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013138429696, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:703.090 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV success controller/src/xmpp/xmpp_connection.cc 402 2025-06-02 Mon 02:08:24:704.415 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:704.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104704390 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:704.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:704.534 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:704.569 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1/dns:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:24:704.573 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830104704511 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenSent next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:704.680 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:704.696 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:704.708 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830104704551 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Sent Start Tls, OpenConfirm Feature Negotiation event = EvStreamFeatureRequest ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:704.820 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:706.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1/dns:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:24:706.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:706.162 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830104706089 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Trigger Client Ssl Handshake event = EvTlsProceed identifier = network-dns@contrailsystems.com ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:706.170 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:706.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:706.194 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830104706149 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:706.199 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:706.206 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:706.228 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830104706198 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:dns-server:10.0.0.129, Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:24:706.240 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:24:710.695 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV success controller/src/xmpp/xmpp_connection.cc 402 2025-06-02 Mon 02:08:24:710.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:710.936 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: SANDESH: Send FAILED: 1748830104710831 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:711.002 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013142628096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:713.068 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:713.109 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104713044 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:24:713.122 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:24:713.131 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:24:713.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830104713093 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:24:713.188 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:25:238.842 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36524 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:08:25:285.810 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013316929856, Pid 1165409]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36524 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:08:25:602.460 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:25:602.574 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:25:602.611 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:08:25:602.698 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Idle 2025-06-02 Mon 02:08:25:602.789 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:25:602.814 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830105602642 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:25:602.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:25:602.829 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:08:25:602.858 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:25:602.988 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830105602776 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:25:603.009 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:26:602.816 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:08:26:603.013 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:26:603.093 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:26:603.118 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830106602883 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:26:603.126 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:26:603.134 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:26:603.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830106603081 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:26:603.302 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:28:603.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:28:603.262 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:28:603.271 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:08:28:603.384 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Idle 2025-06-02 Mon 02:08:28:603.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:28:603.554 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830108603297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:28:603.571 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:28:603.586 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:08:28:603.627 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:28:603.782 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830108603506 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:28:603.803 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:28:883.301 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:28:883.375 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830108883268 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:28:883.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:28:887.820 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:28:887.868 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830108887768 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:28:887.877 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:29:218.784 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:29:218.835 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: SANDESH: Send FAILED: 1748830109218751 AgentVn [SYS_INFO]: VnObjectLog: vn= [ event = Addition uuid = e321a686-481b-40eb-913d-c368c1a2c12e name = default-domain:default-project:ip-fabric acl_uuid = 9f2327dc-c527-41a5-8345-983201b80262 vrf = default-domain:default-project:ip-fabric:ip-fabric ipv4_forwarding = 1 bridging = 1 admin_state = 1 ] file = controller/src/vnsw/agent/oper/vn.cc line = 1391 2025-06-02 Mon 02:08:29:218.848 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013155223296, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: VnObjectLog 2025-06-02 Mon 02:08:29:225.237 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:29:225.293 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: SANDESH: Send FAILED: 1748830109225216 AgentVn [SYS_INFO]: VnObjectLog: vn= [ event = Modification uuid = e321a686-481b-40eb-913d-c368c1a2c12e name = default-domain:default-project:ip-fabric acl_uuid = 9f2327dc-c527-41a5-8345-983201b80262 vrf = default-domain:default-project:ip-fabric:ip-fabric ipv4_forwarding = 1 bridging = 1 admin_state = 1 ] file = controller/src/vnsw/agent/oper/vn.cc line = 1391 2025-06-02 Mon 02:08:29:225.306 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013163620096, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: VnObjectLog 2025-06-02 Mon 02:08:29:297.715 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:29:297.782 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830109297689 AgentVn [SYS_INFO]: VnObjectLog: vn= [ event = Modification uuid = e321a686-481b-40eb-913d-c368c1a2c12e name = default-domain:default-project:ip-fabric acl_uuid = 9f2327dc-c527-41a5-8345-983201b80262 vrf = default-domain:default-project:ip-fabric:ip-fabric ipv4_forwarding = 1 bridging = 1 admin_state = 1 ] file = controller/src/vnsw/agent/oper/vn.cc line = 1391 2025-06-02 Mon 02:08:29:297.802 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Idle for event: EvSandeshSend message: VnObjectLog 2025-06-02 Mon 02:08:30:603.522 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:08:30:603.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:30:603.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:30:603.972 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830110603624 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:30:603.983 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:30:603.995 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:30:604.166 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: SANDESH: Send FAILED: 1748830110603922 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1165409 http_port = 8085 start_time = 1748830104599173 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:30:604.188 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140013151024896, Pid 1165409]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:44:958.666 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: vhost interface name : vhost0 2025-06-02 Mon 02:08:44:958.755 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: vhost IP Address : 10.0.0.129/24 2025-06-02 Mon 02:08:44:958.763 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: vhost gateway : 0.0.0.0 2025-06-02 Mon 02:08:44:958.768 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Ethernet ports : ens3 2025-06-02 Mon 02:08:44:958.773 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Loopback IP : 0.0.0.0 2025-06-02 Mon 02:08:44:958.778 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Servers : 10.0.0.129:5269 2025-06-02 Mon 02:08:44:958.785 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Authentication : 1 2025-06-02 Mon 02:08:44:958.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Server Certificate : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:44:958.794 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Server Key : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:44:958.799 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp CA Certificate : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:44:958.804 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Cluster-Name : 2025-06-02 Mon 02:08:44:958.809 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: DNS Servers : 10.0.0.129:53 2025-06-02 Mon 02:08:44:958.814 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: DNS client port : 8098 2025-06-02 Mon 02:08:44:958.823 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: DNS timeout : 3000 2025-06-02 Mon 02:08:44:958.827 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: DNS max retries : 2 2025-06-02 Mon 02:08:44:958.832 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Dns Authentication : 1 2025-06-02 Mon 02:08:44:958.836 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Server Certificate : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:44:958.840 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp Server Key : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:44:958.844 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Xmpp CA Certificate : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:44:958.849 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TSN Servers : [] 2025-06-02 Mon 02:08:44:958.854 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: COLLECTOR Servers : 10.0.0.129:8086 2025-06-02 Mon 02:08:44:958.858 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Tunnel-Type : MPLSoGRE 2025-06-02 Mon 02:08:44:958.930 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Metadata-Proxy Shared Secret: 8bbe7903-8b3e-4e94-9237-98c3349d0bce 2025-06-02 Mon 02:08:44:958.937 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Metadata-Proxy Port : 8097 2025-06-02 Mon 02:08:44:958.941 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Metadata-Proxy SSL Flag : 0 2025-06-02 Mon 02:08:44:958.971 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Max Vm Flows : 100 2025-06-02 Mon 02:08:44:958.977 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Linklocal Max System Flows : 16384 2025-06-02 Mon 02:08:44:958.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Linklocal Max Vm Flows : 16384 2025-06-02 Mon 02:08:44:958.985 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow cache timeout : 0 2025-06-02 Mon 02:08:44:958.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Stale Interface cleanup timeout : 60 2025-06-02 Mon 02:08:44:958.994 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow thread count : 1 2025-06-02 Mon 02:08:44:958.998 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow latency limit : 0 2025-06-02 Mon 02:08:44:959.002 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow index-mgr sm log count : 0 2025-06-02 Mon 02:08:44:959.007 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow add-tokens : 50 2025-06-02 Mon 02:08:44:959.011 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow ksync-tokens : 25 2025-06-02 Mon 02:08:44:959.016 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow del-tokens : 16 2025-06-02 Mon 02:08:44:959.020 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow update-tokens : 16 2025-06-02 Mon 02:08:44:959.024 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Pin flow netlink task to CPU: 2025-06-02 Mon 02:08:44:959.028 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Maximum sessions : 100 2025-06-02 Mon 02:08:44:959.037 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Maximum session aggregates : 8 2025-06-02 Mon 02:08:44:959.042 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Maximum session endpoints : 5 2025-06-02 Mon 02:08:44:959.046 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Fabric SNAT hash table size : 4096 2025-06-02 Mon 02:08:44:959.050 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Flow excluding Router ID in hash :0 2025-06-02 Mon 02:08:44:959.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Agent Mode : Vrouter 2025-06-02 Mon 02:08:44:959.058 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Gateway Mode : None 2025-06-02 Mon 02:08:44:959.063 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: DHCP Relay Mode : 0 2025-06-02 Mon 02:08:44:959.067 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance netns cmd : /usr/local/bin/opencontrail-vrouter-netns 2025-06-02 Mon 02:08:44:959.072 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance docker cmd : /usr/local/bin/opencontrail-vrouter-docker 2025-06-02 Mon 02:08:44:959.076 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance workers : 0 2025-06-02 Mon 02:08:44:959.080 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance timeout : 0 2025-06-02 Mon 02:08:44:959.085 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance lb ssl : 2025-06-02 Mon 02:08:44:959.089 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Service instance lbaas auth : 2025-06-02 Mon 02:08:44:959.093 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Bgp as a service port range : 50000-50512 2025-06-02 Mon 02:08:44:959.098 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Services queue limit : 1024 2025-06-02 Mon 02:08:44:959.102 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: BGPAAS max shared sessions for service port : 4 2025-06-02 Mon 02:08:44:959.106 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Sandesh Key file : /etc/contrail/ssl/private/server-privkey.pem 2025-06-02 Mon 02:08:44:959.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Sandesh Cert file : /etc/contrail/ssl/certs/server.pem 2025-06-02 Mon 02:08:44:959.114 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Sandesh CA Cert : /etc/contrail/ssl/certs/ca-cert.pem 2025-06-02 Mon 02:08:44:959.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Sandesh SSL Enable : 1 2025-06-02 Mon 02:08:44:959.123 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Introspect SSL Enable : 1 2025-06-02 Mon 02:08:44:959.127 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Hypervisor mode : kvm 2025-06-02 Mon 02:08:44:959.222 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Ethernet Port Encap Type : 2025-06-02 Mon 02:08:44:959.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: Platform mode : Vrouter on host linux kernel 2025-06-02 Mon 02:08:44:959.444 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: VRouter [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"nightly","build-time":"2025-06-02 00:26:11.811431","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-9311-1.novalocal","build-id":"unknown","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-06-02 Mon 02:08:44:959.588 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Logging: DISABLED -> ENABLED 2025-06-02 Mon 02:08:44:959.623 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-06-02 Mon 02:08:44:968.347 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.379 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> self 2025-06-02 Mon 02:08:44:968.386 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:44:968.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:44:968.399 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.403 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-access-control-list -> self 2025-06-02 Mon 02:08:44:968.409 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap access-control-list : self -> virtual-network-access-control-list 2025-06-02 Mon 02:08:44:968.413 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap access-control-list : virtual-network-access-control-list -> nil 2025-06-02 Mon 02:08:44:968.419 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.424 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> self 2025-06-02 Mon 02:08:44:968.428 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:44:968.433 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> virtual-network-network-ipam 2025-06-02 Mon 02:08:44:968.438 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:44:968.443 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.447 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-qos-config -> self 2025-06-02 Mon 02:08:44:968.453 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-config : self -> virtual-network-qos-config 2025-06-02 Mon 02:08:44:968.457 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-config : virtual-network-qos-config -> nil 2025-06-02 Mon 02:08:44:968.462 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.467 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-security-logging-object -> self 2025-06-02 Mon 02:08:44:968.480 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : self -> virtual-network-security-logging-object 2025-06-02 Mon 02:08:44:968.484 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : virtual-network-security-logging-object -> nil 2025-06-02 Mon 02:08:44:968.490 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.494 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : logical-router-virtual-network -> self 2025-06-02 Mon 02:08:44:968.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router-virtual-network : self -> logical-router-virtual-network 2025-06-02 Mon 02:08:44:968.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router-virtual-network : logical-router-virtual-network -> logical-router-virtual-network 2025-06-02 Mon 02:08:44:968.509 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router : self -> logical-router-virtual-network 2025-06-02 Mon 02:08:44:968.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router : logical-router-virtual-network -> nil 2025-06-02 Mon 02:08:44:968.519 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-network 2025-06-02 Mon 02:08:44:968.523 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-multicast-policy -> self 2025-06-02 Mon 02:08:44:968.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap multicast-policy : self -> virtual-network-multicast-policy 2025-06-02 Mon 02:08:44:968.532 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap multicast-policy : virtual-network-multicast-policy -> nil 2025-06-02 Mon 02:08:44:968.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for routing-instance 2025-06-02 Mon 02:08:44:968.542 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> self 2025-06-02 Mon 02:08:44:968.547 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:44:968.552 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-routing-instance 2025-06-02 Mon 02:08:44:968.556 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:44:968.561 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:44:968.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for security-group 2025-06-02 Mon 02:08:44:968.571 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-group : security-group-access-control-list -> self 2025-06-02 Mon 02:08:44:968.576 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap access-control-list : self -> security-group-access-control-list 2025-06-02 Mon 02:08:44:968.583 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap access-control-list : security-group-access-control-list -> nil 2025-06-02 Mon 02:08:44:968.589 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for tag 2025-06-02 Mon 02:08:44:968.606 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> self 2025-06-02 Mon 02:08:44:968.612 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:968.616 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : policy-management-application-policy-set -> application-policy-set-tag 2025-06-02 Mon 02:08:44:968.622 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap policy-management : policy-management-application-policy-set -> nil 2025-06-02 Mon 02:08:44:968.627 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.631 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:44:968.636 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:968.640 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:968.645 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:44:968.649 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:44:968.655 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.660 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-06-02 Mon 02:08:44:968.664 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:44:968.669 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : virtual-machine-interface-virtual-machine -> nil 2025-06-02 Mon 02:08:44:968.674 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.678 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-06-02 Mon 02:08:44:968.682 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:44:968.687 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> nil 2025-06-02 Mon 02:08:44:968.692 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.698 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:44:968.703 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:968.707 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:968.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:44:968.716 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:44:968.722 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.726 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-routing-instance -> self 2025-06-02 Mon 02:08:44:968.731 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-routing-instance : self -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:44:968.736 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-routing-instance : virtual-machine-interface-routing-instance -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:44:968.741 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : self -> virtual-machine-interface-routing-instance 2025-06-02 Mon 02:08:44:968.745 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : virtual-machine-interface-routing-instance -> nil 2025-06-02 Mon 02:08:44:968.751 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.755 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-group -> self 2025-06-02 Mon 02:08:44:968.759 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-group : self -> virtual-machine-interface-security-group 2025-06-02 Mon 02:08:44:968.764 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-group : virtual-machine-interface-security-group -> nil 2025-06-02 Mon 02:08:44:968.798 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.858 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : alias-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:968.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap alias-ip : self -> alias-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.869 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap alias-ip : alias-ip-pool-alias-ip -> alias-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.874 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap alias-ip-pool : virtual-network-alias-ip-pool -> alias-ip-pool-alias-ip 2025-06-02 Mon 02:08:44:968.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-alias-ip-pool 2025-06-02 Mon 02:08:44:968.887 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-alias-ip-pool -> nil 2025-06-02 Mon 02:08:44:968.893 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.898 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:968.903 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.907 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.913 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-06-02 Mon 02:08:44:968.917 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:44:968.922 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:44:968.927 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-06-02 Mon 02:08:44:968.931 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-06-02 Mon 02:08:44:968.936 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.941 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:968.945 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.950 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.954 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-06-02 Mon 02:08:44:968.958 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:44:968.963 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-floating-ip-pool 2025-06-02 Mon 02:08:44:968.967 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:44:968.972 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:44:968.980 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:968.984 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:968.988 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.993 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap floating-ip : instance-ip-floating-ip -> floating-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:968.998 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap instance-ip : instance-ip-virtual-network -> instance-ip-floating-ip 2025-06-02 Mon 02:08:44:969.002 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> instance-ip-virtual-network 2025-06-02 Mon 02:08:44:969.007 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : instance-ip-virtual-network -> nil 2025-06-02 Mon 02:08:44:969.012 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : instance-ip-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap instance-ip : self -> instance-ip-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap instance-ip : instance-ip-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:44:969.031 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.040 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-route-table -> self 2025-06-02 Mon 02:08:44:969.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap interface-route-table : self -> virtual-machine-interface-route-table 2025-06-02 Mon 02:08:44:969.049 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap interface-route-table : virtual-machine-interface-route-table -> nil 2025-06-02 Mon 02:08:44:969.055 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.059 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : subnet-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap subnet : self -> subnet-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.069 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap subnet : subnet-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:44:969.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.079 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.088 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> logical-interface-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.094 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-06-02 Mon 02:08:44:969.099 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:44:969.104 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.114 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-port-group-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> virtual-port-group-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.124 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : physical-router-physical-interface -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:44:969.129 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:44:969.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.143 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.147 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.152 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-as-a-service : bgpaas-bgp-router -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.157 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-router : self -> bgpaas-bgp-router 2025-06-02 Mon 02:08:44:969.162 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-router : instance-bgp-router -> bgpaas-bgp-router 2025-06-02 Mon 02:08:44:969.166 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : self -> instance-bgp-router 2025-06-02 Mon 02:08:44:969.171 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : instance-bgp-router -> nil 2025-06-02 Mon 02:08:44:969.176 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.187 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.192 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-as-a-service : bgpaas-health-check -> bgpaas-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-health-check : self -> bgpaas-health-check 2025-06-02 Mon 02:08:44:969.202 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-health-check : bgpaas-health-check -> nil 2025-06-02 Mon 02:08:44:969.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.211 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-qos-config -> self 2025-06-02 Mon 02:08:44:969.216 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-config : self -> virtual-machine-interface-qos-config 2025-06-02 Mon 02:08:44:969.220 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-config : virtual-machine-interface-qos-config -> nil 2025-06-02 Mon 02:08:44:969.226 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.230 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-logging-object -> self 2025-06-02 Mon 02:08:44:969.235 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : self -> virtual-machine-interface-security-logging-object 2025-06-02 Mon 02:08:44:969.239 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : virtual-machine-interface-security-logging-object -> nil 2025-06-02 Mon 02:08:44:969.245 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.249 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-06-02 Mon 02:08:44:969.255 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.265 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.269 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.274 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:44:969.278 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-06-02 Mon 02:08:44:969.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.290 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-06-02 Mon 02:08:44:969.294 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.299 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.303 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-06-02 Mon 02:08:44:969.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : virtual-machine-interface-bridge-domain -> nil 2025-06-02 Mon 02:08:44:969.313 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.318 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-tag -> self 2025-06-02 Mon 02:08:44:969.347 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> virtual-machine-interface-tag 2025-06-02 Mon 02:08:44:969.352 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-interface-tag 2025-06-02 Mon 02:08:44:969.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.362 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.367 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.372 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.381 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:969.388 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.392 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-06-02 Mon 02:08:44:969.397 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:969.402 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-tag -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:969.409 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> virtual-network-tag 2025-06-02 Mon 02:08:44:969.414 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> virtual-network-tag 2025-06-02 Mon 02:08:44:969.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.423 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.427 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.431 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.436 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:969.446 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.450 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-06-02 Mon 02:08:44:969.455 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:44:969.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:44:969.464 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-06-02 Mon 02:08:44:969.468 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-06-02 Mon 02:08:44:969.473 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.477 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.481 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.486 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.490 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.494 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:969.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.507 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-06-02 Mon 02:08:44:969.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:44:969.516 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> virtual-machine-interface-sub-interface 2025-06-02 Mon 02:08:44:969.521 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:44:969.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-06-02 Mon 02:08:44:969.530 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-06-02 Mon 02:08:44:969.534 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-06-02 Mon 02:08:44:969.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.543 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.547 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.551 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.556 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.560 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:969.566 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.570 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : project-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.579 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap project : self -> project-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.584 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap project : project-tag -> project-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.589 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> project-tag 2025-06-02 Mon 02:08:44:969.603 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : application-policy-set-tag -> project-tag 2025-06-02 Mon 02:08:44:969.610 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.615 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-06-02 Mon 02:08:44:969.619 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.624 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.628 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:969.632 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:969.638 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.643 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> self 2025-06-02 Mon 02:08:44:969.648 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.652 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.657 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.662 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> nil 2025-06-02 Mon 02:08:44:969.667 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-machine-interface 2025-06-02 Mon 02:08:44:969.671 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : logical-router-interface -> self 2025-06-02 Mon 02:08:44:969.676 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router : self -> logical-router-interface 2025-06-02 Mon 02:08:44:969.681 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-router : logical-router-interface -> nil 2025-06-02 Mon 02:08:44:969.686 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for physical-interface 2025-06-02 Mon 02:08:44:969.691 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : physical-router-physical-interface -> self 2025-06-02 Mon 02:08:44:969.695 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:44:969.700 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.705 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for physical-interface 2025-06-02 Mon 02:08:44:969.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : virtual-port-group-physical-interface -> self 2025-06-02 Mon 02:08:44:969.718 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-port-group-physical-interface : self -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:44:969.722 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-port-group-physical-interface : virtual-port-group-physical-interface -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:44:969.727 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-port-group : self -> virtual-port-group-physical-interface 2025-06-02 Mon 02:08:44:969.731 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.736 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:44:969.741 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> self 2025-06-02 Mon 02:08:44:969.745 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : self -> physical-interface-logical-interface 2025-06-02 Mon 02:08:44:969.750 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-06-02 Mon 02:08:44:969.754 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:44:969.758 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.763 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:44:969.767 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-interface : logical-interface-virtual-machine-interface -> self 2025-06-02 Mon 02:08:44:969.772 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> logical-interface-virtual-machine-interface 2025-06-02 Mon 02:08:44:969.777 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> nil 2025-06-02 Mon 02:08:44:969.782 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for logical-interface 2025-06-02 Mon 02:08:44:969.786 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap logical-interface : physical-router-logical-interface -> self 2025-06-02 Mon 02:08:44:969.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : self -> physical-router-logical-interface 2025-06-02 Mon 02:08:44:969.795 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-logical-interface -> nil 2025-06-02 Mon 02:08:44:969.800 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for physical-router 2025-06-02 Mon 02:08:44:969.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-router : physical-router-physical-interface -> self 2025-06-02 Mon 02:08:44:969.811 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : self -> physical-router-physical-interface 2025-06-02 Mon 02:08:44:969.816 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap physical-interface : physical-router-physical-interface -> nil 2025-06-02 Mon 02:08:44:969.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for service-health-check 2025-06-02 Mon 02:08:44:969.826 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-06-02 Mon 02:08:44:969.831 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-06-02 Mon 02:08:44:969.836 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> service-port-health-check 2025-06-02 Mon 02:08:44:969.840 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:969.845 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:44:969.849 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:44:969.855 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for service-health-check 2025-06-02 Mon 02:08:44:969.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-06-02 Mon 02:08:44:969.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-06-02 Mon 02:08:44:969.869 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> service-port-health-check 2025-06-02 Mon 02:08:44:969.873 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.878 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.887 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> port-tuple-interface 2025-06-02 Mon 02:08:44:969.891 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-06-02 Mon 02:08:44:969.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-06-02 Mon 02:08:44:969.900 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-06-02 Mon 02:08:44:969.908 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for qos-config 2025-06-02 Mon 02:08:44:969.912 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-config : global-qos-config-qos-config -> self 2025-06-02 Mon 02:08:44:969.918 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap global-qos-config : global-qos-config-qos-config -> nil 2025-06-02 Mon 02:08:44:969.924 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for forwarding-class 2025-06-02 Mon 02:08:44:969.928 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap forwarding-class : forwarding-class-qos-queue -> self 2025-06-02 Mon 02:08:44:969.934 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-queue : self -> forwarding-class-qos-queue 2025-06-02 Mon 02:08:44:969.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap qos-queue : forwarding-class-qos-queue -> nil 2025-06-02 Mon 02:08:44:969.944 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:44:969.948 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : security-logging-object-network-policy -> self 2025-06-02 Mon 02:08:44:969.957 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap network-policy : self -> security-logging-object-network-policy 2025-06-02 Mon 02:08:44:969.962 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap network-policy : security-logging-object-network-policy -> nil 2025-06-02 Mon 02:08:44:969.967 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:44:969.971 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : security-logging-object-security-group -> self 2025-06-02 Mon 02:08:44:969.976 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-group : self -> security-logging-object-security-group 2025-06-02 Mon 02:08:44:969.980 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-group : security-logging-object-security-group -> nil 2025-06-02 Mon 02:08:44:969.985 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:44:969.990 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : firewall-policy-security-logging-object -> self 2025-06-02 Mon 02:08:44:969.995 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-security-logging-object : self -> firewall-policy-security-logging-object 2025-06-02 Mon 02:08:44:970.000 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-security-logging-object : firewall-policy-security-logging-object -> firewall-policy-security-logging-object 2025-06-02 Mon 02:08:44:970.004 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : firewall-policy-security-logging-object -> nil 2025-06-02 Mon 02:08:44:970.010 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for security-logging-object 2025-06-02 Mon 02:08:44:970.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap security-logging-object : firewall-rule-security-logging-object -> self 2025-06-02 Mon 02:08:44:970.022 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule-security-logging-object : self -> firewall-rule-security-logging-object 2025-06-02 Mon 02:08:44:970.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule-security-logging-object : firewall-rule-security-logging-object -> firewall-rule-security-logging-object 2025-06-02 Mon 02:08:44:970.032 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : firewall-rule-security-logging-object -> nil 2025-06-02 Mon 02:08:44:970.051 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for virtual-router 2025-06-02 Mon 02:08:44:970.055 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-router : virtual-router-sub-cluster -> self 2025-06-02 Mon 02:08:44:970.061 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap sub-cluster : self -> virtual-router-sub-cluster 2025-06-02 Mon 02:08:44:970.065 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap sub-cluster : bgp-router-sub-cluster -> virtual-router-sub-cluster 2025-06-02 Mon 02:08:44:970.070 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-router : self -> bgp-router-sub-cluster 2025-06-02 Mon 02:08:44:970.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bgp-router : bgp-router-sub-cluster -> nil 2025-06-02 Mon 02:08:44:970.082 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for bridge-domain 2025-06-02 Mon 02:08:44:970.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-06-02 Mon 02:08:44:970.219 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:44:970.229 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:44:970.235 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-06-02 Mon 02:08:44:970.240 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-06-02 Mon 02:08:44:970.245 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for bridge-domain 2025-06-02 Mon 02:08:44:970.250 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-06-02 Mon 02:08:44:970.254 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-06-02 Mon 02:08:44:970.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-06-02 Mon 02:08:44:970.265 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for application-policy-set 2025-06-02 Mon 02:08:44:970.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> self 2025-06-02 Mon 02:08:44:970.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:970.282 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:970.286 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-06-02 Mon 02:08:44:970.291 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-06-02 Mon 02:08:44:970.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:44:970.301 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:44:970.307 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.316 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.321 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : firewall-rule-service-group -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.326 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-group : self -> firewall-rule-service-group 2025-06-02 Mon 02:08:44:970.332 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap service-group : firewall-rule-service-group -> nil 2025-06-02 Mon 02:08:44:970.337 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:44:970.342 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:44:970.346 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.351 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.383 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : firewall-rule-address-group -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.388 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap address-group : self -> firewall-rule-address-group 2025-06-02 Mon 02:08:44:970.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap address-group : address-group-tag -> firewall-rule-address-group 2025-06-02 Mon 02:08:44:970.402 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> address-group-tag 2025-06-02 Mon 02:08:44:970.407 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : address-group-tag -> nil 2025-06-02 Mon 02:08:44:970.412 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:44:970.417 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:44:970.421 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.430 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : firewall-rule-tag -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.455 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : self -> firewall-rule-tag 2025-06-02 Mon 02:08:44:970.461 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap tag : firewall-rule-tag -> nil 2025-06-02 Mon 02:08:44:970.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Updating dependency tacker rules for firewall-policy 2025-06-02 Mon 02:08:44:970.470 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-06-02 Mon 02:08:44:970.475 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.479 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.483 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-06-02 Mon 02:08:44:970.488 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Adding ReactorMap firewall-rule : firewall-policy-firewall-rule -> nil 2025-06-02 Mon 02:08:45:155.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.0.0.129:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:45:156.145 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Current receive sock buffer size is 262144 2025-06-02 Mon 02:08:45:156.253 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Vrouter family is 34 2025-06-02 Mon 02:08:45:156.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535328487168, Pid 1167645]: KsyncTxQueue CPU pinning policy <>. KsyncTxQueuen not pinned to CPU 2025-06-02 Mon 02:08:45:156.523 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Flow table size : 20185088 2025-06-02 Mon 02:08:45:178.307 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Flow table size : 161218560 2025-06-02 Mon 02:08:45:178.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SetHugePages: Bridge table size:20185088 Flow table size:161218560 2025-06-02 Mon 02:08:45:178.371 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Mem mapping hugepage file:/dev/hugepages2M/bridge size:20185088 2025-06-02 Mon 02:08:45:178.405 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Mem mapped hugepage file:/dev/hugepages2M/bridge to addr:0x7fd0ec400000 2025-06-02 Mon 02:08:45:178.413 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Mem mapping hugepage file:/dev/hugepages2M/flow size:161218560 2025-06-02 Mon 02:08:45:178.419 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Mem mapped hugepage file:/dev/hugepages2M/flow to addr:0x7fd0e2a00000 2025-06-02 Mon 02:08:45:178.432 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Sending Huge Page configuration to VROUTER 2025-06-02 Mon 02:08:45:178.502 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Vrouter using already allocated memory 2025-06-02 Mon 02:08:45:192.693 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Flow table size : 161218560 2025-06-02 Mon 02:08:45:192.844 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Flow table size : 20185088 2025-06-02 Mon 02:08:45:279.041 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: ROLE : Generator 2025-06-02 Mon 02:08:45:279.072 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: MODULE : contrail-vrouter-agent 2025-06-02 Mon 02:08:45:279.077 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: SOURCE : cn-jenkins-deploy-platform-juju-os-542-1 2025-06-02 Mon 02:08:45:279.082 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: NODE TYPE : Compute 2025-06-02 Mon 02:08:45:279.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: INSTANCE ID : 0 2025-06-02 Mon 02:08:45:279.091 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: HTTP SERVER PORT : 8085 2025-06-02 Mon 02:08:45:279.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-06-02 Mon 02:08:45:279.405 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: HTTP Introspect Init 2025-06-02 Mon 02:08:45:279.589 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8085 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:45:279.656 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Sandesh Http Server Port: 8085 2025-06-02 Mon 02:08:45:279.706 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: NOT Writing http_port 8085TO : /tmp/contrail-vrouter-agent.1166574.http_port 2025-06-02 Mon 02:08:45:279.715 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: CONNECT TO COLLECTOR: 1 2025-06-02 Mon 02:08:45:279.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Idle 2025-06-02 Mon 02:08:45:280.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: No Client: 1748830125279926 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:45:280.352 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvStart in state Idle 2025-06-02 Mon 02:08:45:280.755 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Disconnect 2025-06-02 Mon 02:08:45:280.905 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvCollectorUpdate in state Disconnect 2025-06-02 Mon 02:08:45:281.158 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:45:281.247 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:281.284 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280438 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:__default__ index = 0 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-06-02 Mon 02:08:45:281.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-06-02 Mon 02:08:45:281.311 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:281.329 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280486 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:ip-fabric index = 1 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-06-02 Mon 02:08:45:281.338 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-06-02 Mon 02:08:45:281.346 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:281.387 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280743 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:45:281.397 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:281.408 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:281.523 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280892 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:45:281.539 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:45:281.548 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface lo flags 65609 2025-06-02 Mon 02:08:45:281.553 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:282.305 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280951 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:45:282.328 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:45:282.346 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:282.374 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125280971 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:45:282.390 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:282.402 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:282.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125281233 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:45:282.548 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:45:281.587 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface ens3 flags 69699 2025-06-02 Mon 02:08:45:282.738 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface fan-252 flags 69699 2025-06-02 Mon 02:08:45:282.754 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface ftun0 flags 69699 2025-06-02 Mon 02:08:45:283.127 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface lxdbr0 flags 4099 2025-06-02 Mon 02:08:45:283.158 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vethe884fb7d flags 69699 2025-06-02 Mon 02:08:45:283.200 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vethd808c2e3 flags 69699 2025-06-02 Mon 02:08:45:283.205 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth23343b0c flags 69699 2025-06-02 Mon 02:08:45:283.235 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vethcc1cc497 flags 69699 2025-06-02 Mon 02:08:45:283.240 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth9204b2f6 flags 69699 2025-06-02 Mon 02:08:45:283.290 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth5b2dff57 flags 69699 2025-06-02 Mon 02:08:45:283.298 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vethcc87f2d4 flags 69699 2025-06-02 Mon 02:08:45:283.334 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth515b420e flags 69699 2025-06-02 Mon 02:08:45:283.342 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth90da5e1b flags 69699 2025-06-02 Mon 02:08:45:283.378 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth1d674a27 flags 69699 2025-06-02 Mon 02:08:45:283.386 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface veth2c304bf7 flags 69699 2025-06-02 Mon 02:08:45:283.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vethf9f75916 flags 69699 2025-06-02 Mon 02:08:45:283.424 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface docker0 flags 4099 2025-06-02 Mon 02:08:45:283.456 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface pkt1 flags 65601 2025-06-02 Mon 02:08:45:283.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface pkt3 flags 65601 2025-06-02 Mon 02:08:45:283.470 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface pkt2 flags 65601 2025-06-02 Mon 02:08:45:283.484 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface vhost0 flags 69699 2025-06-02 Mon 02:08:45:283.492 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink interface message RTM_NEWLINK for interface pkt0 flags 69699 2025-06-02 Mon 02:08:45:283.562 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface lo 2025-06-02 Mon 02:08:45:283.585 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-06-02 Mon 02:08:45:283.613 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface lxdbr0 2025-06-02 Mon 02:08:45:283.633 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface docker0 2025-06-02 Mon 02:08:45:283.646 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-06-02 Mon 02:08:45:283.676 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface lo 2025-06-02 Mon 02:08:45:283.689 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-06-02 Mon 02:08:45:283.701 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface ftun0 2025-06-02 Mon 02:08:45:283.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-06-02 Mon 02:08:45:283.724 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink address message RTM_NEWADDR for interface pkt0 2025-06-02 Mon 02:08:45:283.789 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:283.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.835 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.0.0.1/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.847 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.0.0.2/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.0.0.4/32 Interface vhost0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.872 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 10.91.248.0/24 Interface lxdbr0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.879 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:283.893 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 172.17.0.0/16 Interface docker0 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.906 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Handle netlink route message RTM_NEWROUTE : 252.0.0.0/8 Interface fan-252 GW 0.0.0.0 2025-06-02 Mon 02:08:45:283.914 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:283.973 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:45:283.980 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:284.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:45:284.335 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:284.339 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:284.344 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:45:284.348 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:284.352 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:45:284.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-06-02 Mon 02:08:45:284.361 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-06-02 Mon 02:08:45:284.385 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:284.390 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:284.394 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:284.398 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:284.403 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-06-02 Mon 02:08:45:284.407 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:45:284.411 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-06-02 Mon 02:08:45:284.415 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-06-02 Mon 02:08:45:284.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-06-02 Mon 02:08:45:284.424 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:45:284.430 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:45:284.436 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-06-02 Mon 02:08:45:284.442 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:45:284.448 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:45:284.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:45:284.467 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:45:284.473 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-06-02 Mon 02:08:45:305.679 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:9091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:45:306.311 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Router ID configured. Connection to controller initiated 2025-06-02 Mon 02:08:45:307.139 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:307.172 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125306806 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:307.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:307.582 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:307.632 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125307234 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.129, Collector connection down, No Configuration for self Number of connections:2, Expected: 3 ], ] ] ] 2025-06-02 Mon 02:08:45:307.641 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:308.472 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:308.496 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125308007 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:308.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:308.859 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:309.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125308538 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.129, XMPP:dns-server:10.0.0.129, Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:45:309.639 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:311.895 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server [fe80::f816:3eff:fefe:bee4%vhost0]:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-06-02 Mon 02:08:45:361.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:361.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125361677 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:361.911 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:361.919 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:361.932 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125361842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect event = Connect Event local_port = 43893 remote_port = 5269 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:361.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:365.050 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:43893::10.0.0.129:5269 < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-06-02 Mon 02:08:45:365.081 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Event: Tcp Connected peer ip: 10.0.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1295 2025-06-02 Mon 02:08:45:365.219 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:365.271 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125365180 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect next_state = OpenSent event = EvTcpConnected local_port = 43893 remote_port = 5269 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:365.285 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:365.296 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:365.313 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125365198 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:365.335 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:365.347 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:365.364 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125365329 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect event = Connect Event local_port = 42709 remote_port = 8093 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:365.372 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:367.761 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:42709::10.0.0.129:8093 < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-06-02 Mon 02:08:45:367.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Event: Tcp Connected peer ip: 10.0.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1295 2025-06-02 Mon 02:08:45:367.929 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:367.949 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830125367909 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = Connect next_state = OpenSent event = EvTcpConnected local_port = 42709 remote_port = 8093 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:367.956 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:368.061 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:368.071 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830125368055 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenSent next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:368.076 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:368.083 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:368.091 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830125368076 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Sent Start Tls, OpenConfirm Feature Negotiation event = EvStreamFeatureRequest ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:368.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:368.102 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:45:370.083 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:370.130 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830125370065 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Trigger Client Ssl Handshake event = EvTlsProceed identifier = network-control@contrailsystems.com ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:370.142 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:370.089 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:45:372.963 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:373.009 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125372940 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenSent next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:373.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:373.032 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:373.047 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125372961 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Sent Start Tls, OpenConfirm Feature Negotiation event = EvStreamFeatureRequest ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:373.057 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:372.976 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1/dns:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:45:375.030 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV success controller/src/xmpp/xmpp_connection.cc 402 2025-06-02 Mon 02:08:45:375.386 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:375.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125375355 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:375.428 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:381.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:381.190 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830125381107 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Trigger Client Ssl Handshake event = EvTlsProceed identifier = network-dns@contrailsystems.com ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:381.202 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:381.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1/dns:10.0.0.129 RECV Mode Client: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2025-06-02 Mon 02:08:45:381.367 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:381.387 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125381348 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:381.396 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:381.477 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:381.511 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125381432 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:dns-server:10.0.0.129, Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:45:381.520 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:385.427 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-542-1:10.0.0.129 RECV success controller/src/xmpp/xmpp_connection.cc 402 2025-06-02 Mon 02:08:45:385.647 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:385.672 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830125385628 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:385.687 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:387.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:387.969 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125387908 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.129 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-542-1 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-06-02 Mon 02:08:45:387.978 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-06-02 Mon 02:08:45:387.985 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:388.008 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830125387962 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self ], ] ] ] 2025-06-02 Mon 02:08:45:388.014 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:432.037 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:432.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830125431956 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:45:432.106 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:471.817 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:471.885 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830125471711 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:45:471.903 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:45:472.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:472.558 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830125471961 AgentVn [SYS_INFO]: VnObjectLog: vn= [ event = Addition uuid = e321a686-481b-40eb-913d-c368c1a2c12e name = default-domain:default-project:ip-fabric acl_uuid = 9f2327dc-c527-41a5-8345-983201b80262 vrf = default-domain:default-project:ip-fabric:ip-fabric ipv4_forwarding = 1 bridging = 1 admin_state = 1 ] file = controller/src/vnsw/agent/oper/vn.cc line = 1391 2025-06-02 Mon 02:08:45:472.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VnObjectLog 2025-06-02 Mon 02:08:45:476.734 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:45:476.812 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830125476711 AgentVn [SYS_INFO]: VnObjectLog: vn= [ event = Modification uuid = e321a686-481b-40eb-913d-c368c1a2c12e name = default-domain:default-project:ip-fabric acl_uuid = 9f2327dc-c527-41a5-8345-983201b80262 vrf = default-domain:default-project:ip-fabric:ip-fabric ipv4_forwarding = 1 bridging = 1 admin_state = 1 ] file = controller/src/vnsw/agent/oper/vn.cc line = 1391 2025-06-02 Mon 02:08:45:476.824 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VnObjectLog 2025-06-02 Mon 02:08:46:281.298 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:46:281.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:46:281.600 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:08:46:281.706 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Idle 2025-06-02 Mon 02:08:46:281.795 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:46:281.833 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830126281640 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:46:281.844 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:46:281.867 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:08:46:281.879 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:46:282.018 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830126281778 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 13, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:46:282.035 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:46:485.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:55294 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:08:46:529.923 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:55294 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:08:47:281.905 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:08:47:282.149 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:47:282.257 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:47:282.290 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830127281983 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:47:282.298 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:47:282.306 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:47:282.522 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830127282246 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:47:282.546 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:47:524.234 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:55310 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:08:49:282.279 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:49:282.358 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:49:282.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:08:49:282.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Idle 2025-06-02 Mon 02:08:49:282.681 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:49:282.721 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830129282427 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:49:282.733 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:49:282.755 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:08:49:282.791 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:49:283.102 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830129282658 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 16, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:49:283.132 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:51:282.644 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:08:51:282.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:51:283.901 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:51:285.120 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830131282733 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:51:285.140 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:51:285.152 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:51:285.296 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830131283876 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 16, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:51:285.320 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:55:283.005 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:55:283.215 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:08:55:283.230 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:08:55:283.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:08:55:283.461 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:55:283.496 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830135283264 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:55:283.506 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:55:283.518 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:08:55:283.551 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:08:55:283.725 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830135283439 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 16, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:55:283.746 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:08:56:936.493 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54146 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:08:56:985.781 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54146 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:08:59:283.462 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:08:59:283.735 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:08:59:283.862 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:59:283.926 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830139283536 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:08:59:283.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:08:59:283.946 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:08:59:284.166 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830139283844 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:08:59:284.196 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:07:283.887 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:09:07:283.969 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:09:07:283.978 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:09:07:284.104 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Idle 2025-06-02 Mon 02:09:07:284.211 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:07:284.237 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830147284050 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:09:07:284.244 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:09:07:284.252 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:09:07:284.279 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:07:284.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830147284198 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:07:284.448 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:14:959.946 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:960.008 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830154959906 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric total_acl_rules = 4 interface_list= [ [ (*_iter66) = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0, ] ] virtualmachine_list= [ [ ] ] acl = default-domain:default-project:ip-fabric:ip-fabric mirror_acl = ] 2025-06-02 Mon 02:09:14:960.023 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:09:14:960.108 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:960.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830154960087 [SYS_NOTICE]: UveVMInterfaceAgentTrace: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 ip_address = 10.0.0.129 virtual_network = default-domain:default-project:ip-fabric mac_address = fa:16:3e:fe:be:e4 label = 17 active = 1 vm_name = l2_active = 1 uuid = 3077a9a4-a1c3-4cfc-8623-a386a3ad95db ip6_address = :: ip6_active = 1 floating_ips= [ [ ] ] vm_uuid = ip4_active = 1 is_health_check_active = 1 health_check_instance_list= [ [ ] ] admin_state = 1 fixed_ip4_list= [ [ (*_iter26) = 10.0.0.129, ] ] fixed_ip6_list= [ [ ] ] vn_uuid = e321a686-481b-40eb-913d-c368c1a2c12e tx_vlan = 65535 rx_vlan = 65535 parent_interface = ens3 vhostuser_mode = 0 port_mirror_enabled = 0 ] 2025-06-02 Mon 02:09:14:960.480 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVMInterfaceAgentTrace 2025-06-02 Mon 02:09:14:960.491 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:960.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830154960098 [SYS_NOTICE]: UVEVMITags: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 app = tier = site = deployment = labels= [ [ ] ] custom_tags= [ [ ] ] ] 2025-06-02 Mon 02:09:14:960.509 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMITags 2025-06-02 Mon 02:09:14:961.380 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:961.775 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154961359 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric associated_fip_count = 0 in_bandwidth_usage = 0 out_bandwidth_usage = 0 vrf_stats_list= [ [ [ name = default-domain:default-project:ip-fabric:ip-fabric diag_packet_count = 0 unknown_unicast_floods = 0 arp_packet_counts= [ from_vm_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] from_physical_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] ] offload_packet_counts= [ gro = 0 ] nh_packet_counts= [ discards = 0 resolves = 0 l3_receives = 0 l2_receives = 0 local_vm_l3_forwards = 0 local_vm_l2_forwards = 0 vrf_translates = 0 ecmp_forwards = 0 tunnel_nh_stats= [ udp_encaps = 0 mpls_over_udp_encaps = 0 mpls_over_gre_encaps = 0 vxlan_encaps = 0 ] comp_nh_stats= [ source_replication_forwards = 0 edge_replication_forwards = 0 local_vm_l3_forwards = 0 total_multicast_forwards = 0 ] ] ], ] ] ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:09:14:961.800 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:09:14:961.815 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:961.876 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154961371 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = __UNKNOWN__ vrf_stats_list= [ [ [ name = __untitled__ diag_packet_count = 0 unknown_unicast_floods = 0 arp_packet_counts= [ from_vm_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] from_physical_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] ] offload_packet_counts= [ gro = 0 ] nh_packet_counts= [ discards = 0 resolves = 0 l3_receives = 0 l2_receives = 0 local_vm_l3_forwards = 0 local_vm_l2_forwards = 0 vrf_translates = 0 ecmp_forwards = 0 tunnel_nh_stats= [ udp_encaps = 0 mpls_over_udp_encaps = 0 mpls_over_gre_encaps = 0 vxlan_encaps = 0 ] comp_nh_stats= [ source_replication_forwards = 0 edge_replication_forwards = 0 local_vm_l3_forwards = 0 total_multicast_forwards = 0 ] ] ], ] ] ] 2025-06-02 Mon 02:09:14:961.900 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:09:14:961.912 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:966.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154961858 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric port_bucket_bmap= [ udp_sport_bitmap= [ [ (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, (*_iter24) = 0, ] ] udp_dport_bitmap= [ [ (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, (*_iter25) = 0, ] ] tcp_sport_bitmap= [ [ (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, (*_iter26) = 0, ] ] tcp_dport_bitmap= [ [ (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, (*_iter27) = 0, ] ] ] raw_if_stats= [ in_pkts = 51 in_bytes = 3686 out_pkts = 55 out_bytes = 3663 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 0 out_bw_usage = 0 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:09:14:966.161 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:09:14:966.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:966.196 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154961891 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:09:14:966.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:09:14:966.227 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:966.245 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154962999 [SYS_NOTICE]: UveVrouterAgent: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 connected_networks= [ [ (*_iter142) = default-domain:default-project:ip-fabric, ] ] vn_count = 1 ] 2025-06-02 Mon 02:09:14:966.255 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-06-02 Mon 02:09:14:966.268 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:14:966.289 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830154963040 [SYS_NOTICE]: UveVrouterAgent: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 interface_list= [ [ (*_iter143) = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0, ] ] error_intf_list= [ [ ] ] no_config_intf_list= [ [ ] ] vmi_count= [ active = 1 ] down_interface_count = 0 ] 2025-06-02 Mon 02:09:14:966.298 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-06-02 Mon 02:09:15:280.385 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:15:280.545 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155280175 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 56 dequeues = 56 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 69 dequeues = 69 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830147284101 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830154966266 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:09:15:280.569 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:15:280.583 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:15:281.040 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155280324 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:15:281.092 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:15:284.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:09:15:284.371 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:09:15:284.484 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:15:284.513 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155284237 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:09:15:284.520 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:09:15:284.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:15:284.750 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155284469 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:15:284.767 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:15:284.844 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:15:284.906 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155284755 [SYS_NOTICE]: UveVrouterAgent: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 xmpp_peer_list= [ [ [ ip = 10.0.0.129 status = 1 setup_time = 1748830125381437 primary = 1 ], ] ] self_ip_list= [ [ (*_iter146) = 10.0.0.129, ] ] build_info = {"build-info":[{"build-version":"nightly","build-time":"2025-06-02 00:26:11.811431","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-9311-1.novalocal","build-id":"unknown","build-number":"unknown"}]} phy_if= [ [ [ name = ens3 mac_address = fa:16:3e:fe:be:e4 ], ] ] vhost_if= [ name = vhost0 mac_address = fa:16:3e:fe:be:e4 ] log_file = /var/log/contrail/vrouter-agent/contrail-vrouter-agent.log config_file = /etc/contrail/contrail-vrouter-agent.conf log_local = 1 log_level = SYS_DEBUG log_category = sandesh_http_port = 8085 dns_servers= [ [ (*_iter149) = 10.0.0.129, ] ] vhost_cfg= [ name = vhost0 ip = 10.0.0.129 ip_prefix_len = 24 gateway = 10.0.0.1 ] eth_name= [ [ (*_iter150) = ens3, ] ] tunnel_type = MPLSoGRE hypervisor = kvm control_ip = 10.0.0.129 dns_server_list_cfg= [ [ (*_iter152) = 10.0.0.129:53, ] ] control_node_list_cfg= [ [ (*_iter153) = 10.0.0.129:5269, ] ] hostname_cfg = cn-jenkins-deploy-platform-juju-os-542-1 flow_cache_timeout_cfg = 0 ll_max_system_flows_cfg = 16384 ll_max_vm_flows_cfg = 16384 max_vm_flows_cfg = 100 gateway_cfg_list= [ [ ] ] headless_mode_cfg = 1 collector_server_list_cfg= [ [ (*_iter155) = 10.0.0.129:8086, ] ] log_flow = 0 mode = VROUTER platform = HOST vr_limits= [ max_labels = 5120 max_nexthops = 524288 max_interfaces = 4355 max_vrfs = 4096 max_mirror_entries = 255 vrouter_max_bridge_entries = 262144 vrouter_max_oflow_bridge_entries = 53248 vrouter_build_info = {"build-info": [{"build-version": "nightly", "build-time": "2025-06-02 00:23:36.867629", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-9311-1.novalocal"}]} vrouter_max_flow_entries = 524288 vrouter_max_oflow_entries = 105472 vrouter_priority_tagging = 1 ] bgpaas_enabled = 0 port_mirror_enabled = 0 subcluster_name = vr_high_watermark = 80 vr_low_watermark = 75 vr_limit_exceeded_list= [ [ ] ] res_limit = 0 res_table_limit = 0 loopback_ip = 0.0.0.0 gateway_list= [ [ (*_iter161) = 10.0.0.1, ] ] ] 2025-06-02 Mon 02:09:15:284.920 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVrouterAgent 2025-06-02 Mon 02:09:15:284.930 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:15:285.068 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155284843 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 51 in_bytes = 3686 out_tpkts = 55 out_bytes = 3663 exception_packets = 27 exception_packets_dropped = 11 exception_packets_allowed = 16 total_flows = 0 aged_flows = 0 uptime = 1748830124959761 phy_if_info= [ [ _iter254->first = ens3 [ speed = -1 duplexity = 255 ], ] ] raw_vhost_stats= [ name = vhost0 in_pkts = 51 in_bytes = 3686 out_pkts = 55 out_bytes = 3663 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 23 in_bytes = 2319 out_pkts = 49 out_bytes = 2940 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:09:15:285.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:09:15:285.144 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:15:285.172 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830155284875 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:09:15:285.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:09:31:284.516 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:09:31:284.611 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:09:31:284.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:09:31:284.889 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Idle 2025-06-02 Mon 02:09:31:284.993 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:31:285.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830171284839 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:09:31:285.025 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:09:31:285.033 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:09:31:285.068 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:31:285.314 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830171284977 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:31:285.335 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:44:963.287 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:44:963.354 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830184963252 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:09:44:963.368 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:09:44:963.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:44:963.552 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830184963401 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 71 in_bytes = 4734 out_pkts = 76 out_bytes = 4673 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 279 out_bw_usage = 269 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:09:44:963.592 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:09:44:963.617 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:44:963.631 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830184963404 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:09:44:963.637 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:09:45:280.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:281.087 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830185280501 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 68 dequeues = 68 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830171284887 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830184963614 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:09:45:281.101 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:45:281.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:281.329 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830185280937 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:45:281.353 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:45:286.001 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:286.095 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830185285658 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 1.01625 five_min_avg = 1.02125 fifteen_min_avg = 0.5925 ] cpu_share = 0.419923 ] ] 2025-06-02 Mon 02:09:45:286.116 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:09:45:286.126 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:286.144 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830185285724 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.419923 used_sys_mem = 4269467568 one_min_cpuload = 1.01625 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:09:45:286.151 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:09:45:286.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:286.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830185285763 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 71 in_bytes = 4734 out_tpkts = 76 out_bytes = 4673 exception_packets = 28 exception_packets_dropped = 12 raw_vhost_stats= [ name = vhost0 in_pkts = 71 in_bytes = 4734 out_pkts = 76 out_bytes = 4673 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 29 in_bytes = 2699 out_pkts = 69 out_bytes = 3988 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:09:45:286.488 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:09:45:286.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:09:45:286.535 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830185285806 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:09:45:286.542 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:09:47:284.984 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:09:47:285.353 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:09:47:285.481 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:47:285.529 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830187285062 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:09:47:285.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:09:47:285.576 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:09:47:285.825 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830187285463 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:09:47:285.851 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:09:48:602.653 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:58336 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:10:14:963.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:14:963.170 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830214963083 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:10:14:963.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:10:14:963.224 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:14:963.286 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830214963217 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 93 in_bytes = 5886 out_pkts = 98 out_bytes = 5717 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 317 out_bw_usage = 288 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:10:14:963.304 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:10:14:963.315 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:14:963.326 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830214963223 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:10:14:963.332 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:10:15:281.828 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:15:282.160 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830215281307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 79 dequeues = 79 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 96 dequeues = 96 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830187285069 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830214963314 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:10:15:282.185 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:15:282.199 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:15:282.612 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830215281759 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 19, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:10:15:282.644 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:15:287.364 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:15:287.550 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830215287312 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 93 in_bytes = 5886 out_tpkts = 98 out_bytes = 5717 exception_packets = 29 exception_packets_allowed = 17 raw_vhost_stats= [ name = vhost0 in_pkts = 93 in_bytes = 5886 out_pkts = 98 out_bytes = 5717 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] total_in_bandwidth_utilization = 5.69206e-16 total_out_bandwidth_utilization = 1.58836e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 105, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 293, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 36 in_bytes = 3113 out_pkts = 91 out_bytes = 5140 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:10:15:287.572 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:10:15:287.581 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:15:287.623 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830215287362 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:10:15:287.630 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:10:17:285.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:10:17:285.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:10:17:285.560 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:10:17:285.662 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Idle 2025-06-02 Mon 02:10:17:285.756 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:17:285.780 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830217285590 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:10:17:285.788 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:10:17:285.796 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:10:17:285.827 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:17:286.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830217285746 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 19, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:10:17:286.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:17:901.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:43896 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:10:17:949.800 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:43896 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:10:28:601.439 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:40480 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:10:28:646.072 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:40480 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:10:44:964.977 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:44:965.058 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830244964938 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:10:44:965.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:10:44:965.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:44:965.240 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830244965110 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 122 in_bytes = 7644 out_pkts = 128 out_bytes = 7899 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 468 out_bw_usage = 581 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:10:44:965.267 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:10:44:965.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:44:965.288 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830244965115 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:10:44:965.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:10:45:282.040 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:282.131 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245281904 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 88 dequeues = 88 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 107 dequeues = 107 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830217285660 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830244965276 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:10:45:282.146 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:45:282.156 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:282.437 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245282011 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:10:45:282.479 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:45:289.657 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:289.777 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245289582 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.52625 five_min_avg = 0.87875 fifteen_min_avg = 0.57 ] cpu_share = 0.4125 ] ] 2025-06-02 Mon 02:10:45:289.789 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:10:45:289.801 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:289.826 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245289626 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.4125 used_sys_mem = 4269467568 one_min_cpuload = 0.52625 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:10:45:289.833 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:10:45:289.840 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:289.985 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245289661 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 122 in_bytes = 7644 out_tpkts = 128 out_bytes = 7899 exception_packets = 31 exception_packets_dropped = 13 exception_packets_allowed = 18 raw_vhost_stats= [ name = vhost0 in_pkts = 122 in_bytes = 7644 out_pkts = 128 out_bytes = 7899 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 51 in_bytes = 4665 out_pkts = 120 out_bytes = 6898 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:10:45:290.006 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:10:45:290.017 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:10:45:290.044 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830245289691 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:10:45:290.050 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:10:47:285.797 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:10:47:286.091 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:10:47:286.215 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:47:286.663 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830247285875 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:10:47:286.694 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:10:47:286.708 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:10:47:286.933 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830247286195 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:10:47:286.955 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:10:49:610.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:35012 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:11:00:295.217 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:53446 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:11:00:341.740 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:53446 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:11:14:965.008 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:14:965.077 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830274964969 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:11:14:965.168 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:11:14:965.190 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:14:965.295 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830274965106 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 148 in_bytes = 9070 out_pkts = 156 out_bytes = 9610 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 393 out_bw_usage = 472 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:11:14:965.330 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:11:14:965.343 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:14:965.361 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830274965110 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:11:14:965.370 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:11:15:282.394 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:15:282.505 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830275282191 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 99 dequeues = 99 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 119 dequeues = 119 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830247285881 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830274965340 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:11:15:282.530 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:15:282.540 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:15:282.778 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830275282346 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 26, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 99, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:11:15:282.798 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:15:289.945 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:15:290.150 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830275289906 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 148 in_bytes = 9070 out_tpkts = 156 out_bytes = 9610 exception_packets = 33 exception_packets_allowed = 20 raw_vhost_stats= [ name = vhost0 in_pkts = 148 in_bytes = 9070 out_pkts = 156 out_bytes = 9610 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] total_in_bandwidth_utilization = 1.90277e-15 total_out_bandwidth_utilization = 2.29851e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 351, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 424, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 64 in_bytes = 5746 out_pkts = 146 out_bytes = 8324 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:11:15:290.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:11:15:290.191 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:15:290.218 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830275289954 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:11:15:290.226 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:11:17:286.239 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:11:17:286.332 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:11:17:286.372 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:11:17:286.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Idle 2025-06-02 Mon 02:11:17:286.758 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:17:286.807 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830277286440 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:11:17:286.818 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:11:17:286.830 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:11:17:286.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:17:287.115 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830277286736 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 26, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 103, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:11:17:287.189 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:44:966.585 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:44:966.697 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830304966547 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:11:44:966.710 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:11:44:966.719 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:44:966.780 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830304966713 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 168 in_bytes = 10138 out_pkts = 177 out_bytes = 10612 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 284 out_bw_usage = 267 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:11:44:966.800 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:11:44:966.807 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:44:966.876 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830304966719 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:11:44:966.883 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:11:45:282.694 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:282.770 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830305282480 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 108 dequeues = 108 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 130 dequeues = 130 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830277286510 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830304966806 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:11:45:282.783 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:45:282.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:283.107 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830305282664 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 26, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 108, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:11:45:283.136 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:45:290.444 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:290.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830305290295 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.51 five_min_avg = 0.8125 fifteen_min_avg = 0.56625 ] cpu_share = 0.4125 ] ] 2025-06-02 Mon 02:11:45:290.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:11:45:290.549 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:290.569 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830305290376 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.4125 used_sys_mem = 4269467568 one_min_cpuload = 0.51 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:11:45:290.576 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:11:45:290.701 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:290.878 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830305290589 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 168 in_bytes = 10138 out_tpkts = 177 out_bytes = 10612 raw_vhost_stats= [ name = vhost0 in_pkts = 168 in_bytes = 10138 out_pkts = 177 out_bytes = 10612 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 70 in_bytes = 6118 out_pkts = 166 out_bytes = 9392 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:11:45:290.910 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:11:45:290.923 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:11:45:290.962 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830305290636 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:11:45:290.970 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:11:47:286.670 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:11:47:286.928 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:11:47:287.042 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:47:287.080 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830307286749 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:11:47:287.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:11:47:287.105 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:11:47:287.325 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830307287024 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 26, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 114, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:11:47:287.348 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:11:50:587.751 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:57732 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:11:50:650.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:57734 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:11:50:694.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:57734 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:12:14:968.129 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:14:968.205 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830334967982 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:12:14:968.220 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:12:14:968.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:14:970.212 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830334968104 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 201 in_bytes = 12227 out_pkts = 210 out_bytes = 13246 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 557 out_bw_usage = 702 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:12:14:970.266 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:12:14:970.285 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:14:970.305 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830334968108 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:12:14:970.314 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:12:15:282.919 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:15:353.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830335282811 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 119 dequeues = 119 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 142 dequeues = 142 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830307286756 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830334970282 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:12:15:353.307 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:15:353.321 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:15:353.550 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830335282890 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 119, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:12:15:353.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:15:353.580 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:15:353.747 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830335290809 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 201 in_bytes = 12227 out_tpkts = 210 out_bytes = 13246 exception_packets = 35 exception_packets_allowed = 22 raw_vhost_stats= [ name = vhost0 in_pkts = 201 in_bytes = 12227 out_pkts = 210 out_bytes = 13246 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] total_in_bandwidth_utilization = 1.71304e-15 total_out_bandwidth_utilization = 2.27682e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 316, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 420, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 88 in_bytes = 8122 out_pkts = 199 out_bytes = 11481 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:12:15:353.761 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:12:15:353.770 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:15:353.803 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830335290842 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:12:15:353.813 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:12:17:288.364 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:12:17:288.417 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:12:17:288.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:12:17:289.300 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Idle 2025-06-02 Mon 02:12:17:289.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:17:289.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830337288453 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:12:17:289.433 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:12:17:289.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:12:17:289.472 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:17:289.700 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830337289380 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 123, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:12:17:289.721 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:44:966.895 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:44:966.964 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830364966860 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:12:44:966.975 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:12:44:966.986 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:44:967.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830364966968 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 323 in_bytes = 22674 out_pkts = 334 out_bytes = 30605 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 2881 out_bw_usage = 4788 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:12:44:967.155 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:12:44:967.169 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:44:967.179 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830364966975 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:12:44:967.185 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:12:45:283.129 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:283.225 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830365282984 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 128 dequeues = 128 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 153 dequeues = 153 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830337289299 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830364967168 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:12:45:283.239 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:45:283.247 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:283.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830365283096 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 128, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:12:45:283.494 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:45:291.478 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:291.546 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830365291423 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.8525 five_min_avg = 0.8625 fifteen_min_avg = 0.59875 ] cpu_share = 0.433333 ] ] 2025-06-02 Mon 02:12:45:291.557 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:12:45:291.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:291.583 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830365291454 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.433333 used_sys_mem = 4269467568 one_min_cpuload = 0.8525 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:12:45:291.590 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:12:45:291.597 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:291.746 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830365291488 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 323 in_bytes = 22674 out_tpkts = 334 out_bytes = 30605 exception_packets = 37 exception_packets_dropped = 14 exception_packets_allowed = 23 raw_vhost_stats= [ name = vhost0 in_pkts = 323 in_bytes = 22674 out_pkts = 334 out_bytes = 30605 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 197 in_bytes = 24851 out_pkts = 321 out_bytes = 21928 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:12:45:291.774 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:12:45:291.781 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:12:45:291.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830365291553 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:12:45:291.814 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:12:47:289.461 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:12:47:289.720 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:12:47:289.870 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:47:289.909 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830367289543 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:12:47:289.917 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:12:47:289.925 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:12:47:290.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830367289813 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 134, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:12:47:290.157 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:12:51:546.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:41842 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:13:14:967.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:14:967.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830394967314 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:13:14:967.447 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:13:14:967.517 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:14:967.629 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830394967495 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 345 in_bytes = 23854 out_pkts = 385 out_bytes = 32895 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 314 out_bw_usage = 610 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:13:14:967.657 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:13:14:967.669 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:14:967.684 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830394967502 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:13:14:967.691 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:13:15:283.409 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:15:283.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830395283275 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 139 dequeues = 139 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 165 dequeues = 165 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830367289552 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830394967667 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:13:15:283.527 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:15:283.536 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:15:283.950 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830395283373 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 139, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:13:15:283.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:15:291.847 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:15:292.238 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830395291815 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 345 in_bytes = 23854 out_tpkts = 385 out_bytes = 32895 exception_packets = 38 exception_packets_allowed = 24 raw_vhost_stats= [ name = vhost0 in_pkts = 345 in_bytes = 23854 out_pkts = 385 out_bytes = 32895 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 5 ] total_in_bandwidth_utilization = 1.32869e-14 total_out_bandwidth_utilization = 8.40257e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 2451, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 1550, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 233 in_bytes = 26511 out_pkts = 343 out_bytes = 23108 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-06-02 Mon 02:13:15:292.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:13:15:292.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:15:292.355 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830395291860 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:13:15:292.365 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:13:17:290.144 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:13:17:290.221 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:13:17:290.230 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:13:17:290.327 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:13:17:290.434 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:17:290.460 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830397290261 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:13:17:290.477 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:13:17:290.484 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:13:17:290.510 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:17:290.781 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830397290424 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 143, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:13:17:290.803 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:44:967.664 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:44:967.726 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830424967594 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:13:44:967.736 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:13:44:967.796 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:44:967.891 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830424967750 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 355 in_bytes = 24454 out_pkts = 427 out_bytes = 34787 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 160 out_bw_usage = 504 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:13:44:967.918 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:13:44:967.929 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:44:967.939 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830424967771 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:13:44:967.946 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:13:45:283.756 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:283.842 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830425283570 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 148 dequeues = 148 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 176 dequeues = 176 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830397290326 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830424967927 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:13:45:283.855 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:45:283.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:284.236 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830425283707 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 148, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:13:45:284.276 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:45:292.173 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:292.251 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830425292120 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.55625 five_min_avg = 0.77125 fifteen_min_avg = 0.58375 ] cpu_share = 0.41875 ] ] 2025-06-02 Mon 02:13:45:292.262 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:13:45:292.271 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:292.286 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830425292156 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.41875 used_sys_mem = 4269467568 one_min_cpuload = 0.55625 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:13:45:292.293 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:13:45:292.299 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:292.424 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830425292194 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 355 in_bytes = 24454 out_tpkts = 427 out_bytes = 34787 raw_vhost_stats= [ name = vhost0 in_pkts = 355 in_bytes = 24454 out_pkts = 427 out_bytes = 34787 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 269 in_bytes = 28151 out_pkts = 353 out_bytes = 23708 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:13:45:292.443 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:13:45:292.451 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:13:45:292.477 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830425292350 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:13:45:292.483 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:13:47:290.460 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:13:47:290.879 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:13:47:291.323 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:47:291.406 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830427290565 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:13:47:291.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:13:47:291.436 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:13:47:291.847 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830427291307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 154, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:13:47:291.876 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:13:52:498.447 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:39298 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:14:14:969.438 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:14:969.523 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830454969402 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:14:14:969.535 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:14:14:969.736 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:14:969.828 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830454969712 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 365 in_bytes = 25150 out_pkts = 452 out_bytes = 36089 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 185 out_bw_usage = 347 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:14:14:969.857 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:14:14:969.866 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:14:969.877 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830454969798 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:14:14:969.884 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:14:15:284.153 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:15:284.262 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830455283894 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 159 dequeues = 159 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 188 dequeues = 188 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830427290572 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830454969865 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:14:15:284.274 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:15:284.282 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:15:284.509 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830455284103 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 31, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 159, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:14:15:285.011 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:15:292.551 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:15:292.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830455292516 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 365 in_bytes = 25150 out_tpkts = 452 out_bytes = 36089 exception_packets = 40 exception_packets_allowed = 26 raw_vhost_stats= [ name = vhost0 in_pkts = 365 in_bytes = 25150 out_pkts = 452 out_bytes = 36089 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 720 out_bandwidth_usage = 573 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] total_in_bandwidth_utilization = 2.03288e-15 total_out_bandwidth_utilization = 9.6494e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 375, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 178, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 291 in_bytes = 29327 out_pkts = 364 out_bytes = 24446 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:14:15:292.850 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:14:15:292.861 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:15:292.890 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830455292554 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:14:15:292.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:14:17:291.220 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:14:17:291.314 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:14:17:291.350 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:14:17:291.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Idle 2025-06-02 Mon 02:14:17:291.588 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:17:291.639 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830457291389 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:14:17:291.657 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:14:17:291.666 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:14:17:291.691 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:17:291.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830457291565 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 31, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 163, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:14:17:291.959 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:44:971.122 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:44:971.187 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830484971088 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:14:44:971.204 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:14:44:971.325 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:44:971.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830484971259 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 372 in_bytes = 25624 out_pkts = 463 out_bytes = 36707 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 126 out_bw_usage = 164 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:14:44:971.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:14:44:971.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:44:971.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830484971269 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:14:44:971.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:14:45:284.555 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:284.704 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830485284368 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 168 dequeues = 168 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 199 dequeues = 199 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830457291461 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830484971510 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:14:45:284.721 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:45:284.731 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:285.020 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830485284506 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 31, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 168, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:14:45:285.066 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:45:292.855 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:292.936 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830485292795 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.34625 five_min_avg = 0.67375 fifteen_min_avg = 0.56125 ] cpu_share = 0.408333 ] ] 2025-06-02 Mon 02:14:45:292.957 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:14:45:292.971 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:292.987 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830485292813 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.408333 used_sys_mem = 4269467568 one_min_cpuload = 0.34625 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:14:45:292.996 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:14:45:293.005 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:293.142 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830485292845 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 372 in_bytes = 25624 out_tpkts = 463 out_bytes = 36707 raw_vhost_stats= [ name = vhost0 in_pkts = 372 in_bytes = 25624 out_pkts = 463 out_bytes = 36707 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 299 in_bytes = 29819 out_pkts = 371 out_bytes = 24920 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:14:45:293.160 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:14:45:293.168 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:14:45:293.222 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830485292877 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:14:45:293.230 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:14:47:291.642 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:14:47:291.979 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:14:47:292.134 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:47:292.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830487291775 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:14:47:292.201 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:14:47:292.213 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:14:47:292.532 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830487292117 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 31, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 174, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:14:47:292.554 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:14:53:454.878 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:35758 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:15:04:702.588 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:38966 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:15:04:745.694 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:38966 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:15:14:971.090 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:14:971.153 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830514970991 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:15:14:971.163 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:15:14:971.173 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:14:971.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830514971138 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 396 in_bytes = 27432 out_pkts = 489 out_bytes = 39037 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 498 out_bw_usage = 642 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:15:14:971.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:15:14:971.306 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:14:971.316 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830514971145 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:15:14:971.323 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:15:15:018.617 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:58188 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:15:15:061.791 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:58188 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:15:15:284.908 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:15:285.041 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830515284719 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 179 dequeues = 179 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 211 dequeues = 211 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830487291783 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830514971304 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:15:15:285.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:15:285.084 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:15:285.350 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830515284832 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 36, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 179, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:15:15:285.480 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:15:293.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:15:293.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830515293079 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 396 in_bytes = 27432 out_tpkts = 489 out_bytes = 39037 exception_packets = 42 exception_packets_allowed = 28 raw_vhost_stats= [ name = vhost0 in_pkts = 396 in_bytes = 27432 out_pkts = 489 out_bytes = 39037 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] total_in_bandwidth_utilization = 1.94614e-15 total_out_bandwidth_utilization = 1.64799e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 359, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 304, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 322 in_bytes = 32023 out_pkts = 395 out_bytes = 26728 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:15:15:293.362 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:15:15:293.380 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:15:293.410 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830515293265 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:15:15:293.417 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:15:17:292.137 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:15:17:292.287 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:15:17:292.324 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:15:17:292.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Idle 2025-06-02 Mon 02:15:17:292.515 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:17:292.541 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830517292363 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:15:17:292.558 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:15:17:292.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:15:17:292.610 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:17:293.000 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830517292502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 36, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 183, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:15:17:293.042 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:25:752.486 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:53258 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:15:25:801.840 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:53258 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:15:44:973.803 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:44:973.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830544973088 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:15:44:973.871 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:15:44:973.881 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:44:973.942 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830544973236 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 410 in_bytes = 28558 out_pkts = 504 out_bytes = 40342 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 300 out_bw_usage = 348 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:15:44:973.962 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:15:44:973.970 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:44:973.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830544973240 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:15:44:973.987 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:15:45:285.060 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:285.138 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830545284941 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 188 dequeues = 188 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 222 dequeues = 222 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830517292418 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830544973969 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:15:45:285.151 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:45:285.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:285.371 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830545285036 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 188, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:15:45:286.070 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:45:293.458 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:293.553 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830545293416 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.36375 five_min_avg = 0.61875 fifteen_min_avg = 0.55 ] cpu_share = 0.406182 ] ] 2025-06-02 Mon 02:15:45:293.566 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:15:45:293.576 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:293.591 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830545293440 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.406182 used_sys_mem = 4269467568 one_min_cpuload = 0.36375 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:15:45:293.599 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:15:45:293.605 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:293.787 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830545293475 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 410 in_bytes = 28558 out_tpkts = 504 out_bytes = 40342 raw_vhost_stats= [ name = vhost0 in_pkts = 410 in_bytes = 28558 out_pkts = 504 out_bytes = 40342 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 334 in_bytes = 33202 out_pkts = 409 out_bytes = 27854 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:15:45:293.821 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:15:45:293.833 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:15:45:293.875 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830545293518 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:15:45:293.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:15:47:292.590 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:15:47:292.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:15:47:292.991 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:47:293.028 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830547292719 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:15:47:293.036 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:15:47:293.044 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:15:47:293.356 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830547292979 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 194, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:15:47:293.384 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:15:54:406.375 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:45236 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:16:14:974.820 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:14:974.895 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830574974777 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:16:14:974.910 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:16:14:975.028 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:14:975.397 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830574974994 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 420 in_bytes = 29194 out_pkts = 515 out_bytes = 40948 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 169 out_bw_usage = 161 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:16:14:975.441 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:16:14:975.456 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:14:975.472 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830574975014 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:16:14:975.481 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:16:15:285.329 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:15:285.500 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830575285200 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 199 dequeues = 199 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 234 dequeues = 234 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830547292728 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830574975453 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:16:15:285.531 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:15:285.544 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:15:285.891 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830575285273 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 39, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 199, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:16:15:285.915 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:15:293.838 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:15:294.044 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830575293800 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 420 in_bytes = 29194 out_tpkts = 515 out_bytes = 40948 exception_packets = 43 exception_packets_allowed = 29 raw_vhost_stats= [ name = vhost0 in_pkts = 420 in_bytes = 29194 out_pkts = 515 out_bytes = 40948 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 12 ] total_in_bandwidth_utilization = 1.19804e-15 total_out_bandwidth_utilization = 1.26852e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 221, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 234, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 342 in_bytes = 33682 out_pkts = 419 out_bytes = 28490 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 7 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 7 ], ] ] ] 2025-06-02 Mon 02:16:15:294.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:16:15:294.085 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:15:294.211 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830575293857 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:16:15:294.223 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:16:17:293.155 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:16:17:293.244 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:16:17:293.286 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:16:17:293.405 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Idle 2025-06-02 Mon 02:16:17:293.526 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:17:293.566 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830577293329 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:16:17:293.591 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:16:17:293.603 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:16:17:293.655 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:17:293.968 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830577293511 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 39, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 203, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:16:17:293.995 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:44:975.155 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:44:975.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830604975125 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:16:44:975.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:16:44:975.309 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:44:975.375 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830604975265 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 536 in_bytes = 39746 out_pkts = 633 out_bytes = 58848 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 2813 out_bw_usage = 4773 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:16:44:975.396 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:16:44:975.404 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:44:975.414 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830604975273 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:16:44:975.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:16:45:285.638 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:285.756 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830605285444 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 208 dequeues = 208 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 245 dequeues = 245 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830577293403 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830604975403 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:16:45:285.770 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:45:285.779 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:286.527 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830605285595 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 39, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 208, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:16:45:286.571 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:45:294.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:294.175 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830605294063 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.785 five_min_avg = 0.6725 fifteen_min_avg = 0.57125 ] cpu_share = 0.370833 ] ] 2025-06-02 Mon 02:16:45:294.186 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:16:45:294.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:294.214 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830605294085 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.370833 used_sys_mem = 4269467568 one_min_cpuload = 0.785 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:16:45:294.220 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:16:45:294.227 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:294.394 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830605294117 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 536 in_bytes = 39746 out_tpkts = 633 out_bytes = 58848 exception_packets = 45 exception_packets_dropped = 15 exception_packets_allowed = 30 raw_vhost_stats= [ name = vhost0 in_pkts = 536 in_bytes = 39746 out_pkts = 633 out_bytes = 58848 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 457 in_bytes = 51456 out_pkts = 535 out_bytes = 39042 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:16:45:294.415 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:16:45:294.423 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:16:45:294.451 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830605294153 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:16:45:294.457 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:16:47:293.547 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:16:47:293.913 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:16:47:294.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:47:294.056 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830607293638 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:16:47:294.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:16:47:294.082 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:16:47:294.405 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830607294013 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 39, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 214, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:16:47:294.431 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:16:49:397.699 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:39072 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:16:49:441.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:39072 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:16:55:509.334 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:50638 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:17:14:975.835 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:14:975.901 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830634975794 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:17:14:975.913 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:17:14:976.186 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:14:976.280 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830634976159 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 550 in_bytes = 40728 out_pkts = 653 out_bytes = 60347 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 261 out_bw_usage = 399 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:17:14:976.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:17:14:976.317 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:14:976.328 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830634976166 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:17:14:976.336 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:17:15:285.874 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:15:285.994 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830635285772 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 219 dequeues = 219 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 257 dequeues = 257 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830607293679 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830634976316 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:17:15:286.023 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:15:286.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:15:286.330 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830635285849 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 42, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 219, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:17:15:286.354 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:15:294.364 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:15:294.708 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830635294323 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 550 in_bytes = 40728 out_tpkts = 653 out_bytes = 60347 exception_packets = 46 exception_packets_allowed = 31 raw_vhost_stats= [ name = vhost0 in_pkts = 550 in_bytes = 40728 out_pkts = 653 out_bytes = 60347 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 1.38344e-14 total_out_bandwidth_utilization = 8.33209e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 2552, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 1537, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 474 in_bytes = 52829 out_pkts = 549 out_bytes = 40024 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:17:15:294.751 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:17:15:294.766 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:15:294.809 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830635294373 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:17:15:294.820 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:17:17:294.040 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:17:17:294.155 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:17:17:294.219 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:17:17:294.389 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Idle 2025-06-02 Mon 02:17:17:294.531 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:17:294.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830637294281 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:17:17:294.579 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:17:17:294.590 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:17:17:294.625 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:17:295.099 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830637294516 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 42, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 223, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:17:17:295.139 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:44:975.533 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:44:976.112 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830664975477 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:17:44:976.137 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:17:44:976.147 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:44:976.217 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830664975630 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 562 in_bytes = 41632 out_pkts = 671 out_bytes = 61507 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 249 out_bw_usage = 320 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:17:44:976.238 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:17:44:976.391 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:44:976.423 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830664975634 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:17:44:976.436 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:17:45:286.165 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:286.251 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830665285981 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 228 dequeues = 228 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 268 dequeues = 268 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830637294387 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830664976246 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:17:45:286.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:45:286.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:286.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830665286115 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 62 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 42, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 228, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:17:45:286.531 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:45:294.562 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:294.636 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830665294528 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.465 five_min_avg = 0.6025 fifteen_min_avg = 0.55375 ] cpu_share = 0.25625 ] ] 2025-06-02 Mon 02:17:45:294.647 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:17:45:294.671 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:294.688 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830665294544 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.25625 used_sys_mem = 4269467568 one_min_cpuload = 0.465 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:17:45:294.694 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:17:45:294.701 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:294.843 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830665294575 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 562 in_bytes = 41632 out_tpkts = 671 out_bytes = 61507 raw_vhost_stats= [ name = vhost0 in_pkts = 562 in_bytes = 41632 out_pkts = 671 out_bytes = 61507 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 489 in_bytes = 53863 out_pkts = 561 out_bytes = 40928 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:17:45:294.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:17:45:294.874 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:17:45:294.900 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830665294608 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:17:45:294.906 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:17:47:294.535 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:17:47:294.795 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:17:47:294.900 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:47:294.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830667294618 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:17:47:294.943 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:17:47:294.951 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:17:47:295.178 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830667294878 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 42, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 234, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:17:47:295.274 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:17:56:454.869 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:34322 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:18:14:976.048 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:14:976.121 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830694975989 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:18:14:976.133 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:18:14:976.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:14:976.359 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830694976161 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 568 in_bytes = 42064 out_pkts = 680 out_bytes = 61985 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 115 out_bw_usage = 127 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:18:14:976.444 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:18:14:976.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:14:976.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830694976170 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:18:14:976.473 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:18:15:286.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:15:286.628 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830695286318 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 239 dequeues = 239 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 280 dequeues = 280 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830667294626 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830694976453 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:18:15:286.651 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:15:286.709 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:15:287.126 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830695286412 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 65 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 32 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 239, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:18:15:287.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:15:294.841 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:15:295.047 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830695294806 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 568 in_bytes = 42064 out_tpkts = 680 out_bytes = 61985 raw_vhost_stats= [ name = vhost0 in_pkts = 568 in_bytes = 42064 out_pkts = 680 out_bytes = 61985 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 9.97466e-16 total_out_bandwidth_utilization = 9.6494e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 184, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 178, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 495 in_bytes = 54215 out_pkts = 567 out_bytes = 41360 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:18:15:295.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:18:15:295.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:15:295.113 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830695294845 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:18:15:295.120 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:18:17:294.903 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:18:17:294.965 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:18:17:294.975 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:18:17:295.094 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Idle 2025-06-02 Mon 02:18:17:295.202 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:17:295.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830697295006 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:18:17:295.251 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:18:17:295.260 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:18:17:295.290 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:17:295.582 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830697295186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 67 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 32 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 243, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:18:17:295.606 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:44:976.346 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:44:976.421 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830724976306 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:18:44:976.438 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:18:44:976.596 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:44:976.720 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830724976580 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 578 in_bytes = 42712 out_pkts = 692 out_bytes = 62665 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 172 out_bw_usage = 181 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:18:44:976.757 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:18:44:976.955 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:44:976.992 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830724976591 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:18:44:977.004 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:18:45:286.785 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:286.896 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725286513 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 248 dequeues = 248 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 291 dequeues = 291 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830697295093 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830724976849 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:18:45:286.916 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:45:286.988 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:287.251 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725286732 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 68 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 32 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 248, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:18:45:287.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:45:295.136 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:295.212 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725295100 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.33125 five_min_avg = 0.54 fifteen_min_avg = 0.535 ] cpu_share = 0.266667 ] ] 2025-06-02 Mon 02:18:45:295.223 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:18:45:295.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:295.250 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725295119 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.266667 used_sys_mem = 4269467568 one_min_cpuload = 0.33125 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:18:45:295.258 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:18:45:295.264 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:295.431 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725295148 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 578 in_bytes = 42712 out_tpkts = 692 out_bytes = 62665 exception_packets = 47 exception_packets_allowed = 32 raw_vhost_stats= [ name = vhost0 in_pkts = 578 in_bytes = 42712 out_pkts = 692 out_bytes = 62665 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 504 in_bytes = 54769 out_pkts = 577 out_bytes = 42008 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:18:45:295.459 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:18:45:295.467 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:18:45:295.494 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830725295189 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:18:45:295.501 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:18:47:295.262 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:18:47:295.580 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:18:47:295.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:47:295.745 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830727295359 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:18:47:295.764 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:18:47:295.772 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:18:47:296.011 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830727295698 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 70 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 32 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 254, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:18:47:296.033 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:18:57:390.860 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54128 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:19:14:976.317 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:14:976.404 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830754976282 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:19:14:976.419 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:19:14:976.603 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:14:976.738 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830754976583 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 584 in_bytes = 43144 out_pkts = 699 out_bytes = 63031 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 115 out_bw_usage = 97 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:19:14:976.791 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:19:14:976.807 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:14:976.825 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830754976590 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:19:14:976.834 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:19:15:287.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:15:287.143 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830755286914 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 259 dequeues = 259 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 303 dequeues = 303 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830727295367 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830754976805 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:19:15:287.157 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:15:287.166 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:15:287.483 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830755286989 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 71 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 259, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:19:15:287.516 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:15:295.618 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:15:295.892 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830755295553 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 584 in_bytes = 43144 out_tpkts = 699 out_bytes = 63031 exception_packets = 48 exception_packets_allowed = 33 raw_vhost_stats= [ name = vhost0 in_pkts = 584 in_bytes = 43144 out_pkts = 699 out_bytes = 63031 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 684 out_bandwidth_usage = 480 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 5.69206e-16 total_out_bandwidth_utilization = 8.07731e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 105, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 149, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 508 in_bytes = 55009 out_pkts = 584 out_bytes = 42482 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:19:15:295.926 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:19:15:295.937 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:15:295.965 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830755295614 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:19:15:295.973 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:19:17:295.734 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:19:17:295.813 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:19:17:295.845 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:19:17:295.942 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:19:17:296.039 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:17:296.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830757295877 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:19:17:296.081 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:19:17:296.089 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:19:17:296.115 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:17:296.350 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830757296028 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 73 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 263, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:19:17:296.374 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:44:978.118 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:44:978.179 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830784978082 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:19:44:978.189 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:19:44:978.308 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:44:978.410 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830784978252 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 594 in_bytes = 43792 out_pkts = 710 out_bytes = 63641 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 172 out_bw_usage = 162 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:19:44:978.442 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:19:44:978.454 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:44:978.469 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830784978277 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:19:44:978.478 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:19:45:287.354 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:287.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830785287149 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 268 dequeues = 268 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 314 dequeues = 314 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830757295941 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830784978452 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:19:45:287.453 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:45:287.465 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:287.732 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830785287307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 74 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 268, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:19:45:287.759 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:45:295.937 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:296.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830785295839 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.305 five_min_avg = 0.495 fifteen_min_avg = 0.51875 ] cpu_share = 0.254167 ] ] 2025-06-02 Mon 02:19:45:296.050 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:19:45:296.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:296.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830785295861 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.254167 used_sys_mem = 4269467568 one_min_cpuload = 0.305 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:19:45:296.095 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:19:45:296.104 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:296.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830785295930 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 594 in_bytes = 43792 out_tpkts = 710 out_bytes = 63641 exception_packets = 49 exception_packets_allowed = 34 raw_vhost_stats= [ name = vhost0 in_pkts = 594 in_bytes = 43792 out_pkts = 710 out_bytes = 63641 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 516 in_bytes = 55493 out_pkts = 594 out_bytes = 43130 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:19:45:296.304 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:19:45:296.315 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:19:45:296.358 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830785295967 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:19:45:296.367 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:19:47:296.112 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:19:47:296.377 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:19:47:296.513 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:47:296.548 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830787296195 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:19:47:296.574 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:19:47:296.585 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:19:47:296.971 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830787296498 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 76 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 274, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:19:47:297.023 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:19:58:351.189 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:43506 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:20:14:979.870 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:14:979.920 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830814979817 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:20:14:979.929 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:20:14:980.020 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:14:980.106 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830814979982 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 600 in_bytes = 44224 out_pkts = 717 out_bytes = 64035 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 115 out_bw_usage = 105 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:20:14:980.128 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:20:14:980.139 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:14:980.150 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830814980000 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:20:14:980.156 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:20:15:287.723 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:15:287.875 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830815287505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 279 dequeues = 279 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 326 dequeues = 326 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830787296206 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830814980138 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:20:15:287.893 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:15:287.905 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:15:288.165 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830815287652 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 77 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 34 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 45, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 279, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:20:15:288.306 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:15:296.225 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:15:296.444 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830815296190 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 600 in_bytes = 44224 out_tpkts = 717 out_bytes = 64035 raw_vhost_stats= [ name = vhost0 in_pkts = 600 in_bytes = 44224 out_pkts = 717 out_bytes = 64035 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 5.42101e-16 total_out_bandwidth_utilization = 7.80626e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 100, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 144, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 520 in_bytes = 55761 out_pkts = 600 out_bytes = 43562 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:20:15:296.478 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:20:15:296.489 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:15:296.517 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830815296239 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:20:15:296.527 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:20:17:097.987 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36950 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:20:17:145.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36950 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:20:17:296.540 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:20:17:296.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:20:17:296.661 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:20:17:296.825 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Idle 2025-06-02 Mon 02:20:17:296.945 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:17:296.974 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830817296719 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:20:17:296.992 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:20:17:297.000 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:20:17:297.043 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:17:297.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830817296932 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 47, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 283, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:20:17:297.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:27:817.296 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:47974 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:20:27:865.665 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:47974 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:20:38:244.633 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:38650 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:20:38:289.920 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:38650 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:20:44:982.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:44:982.451 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830844981654 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 623 in_bytes = 45856 out_pkts = 741 out_bytes = 66372 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 435 out_bw_usage = 623 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:20:44:982.479 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:20:44:982.490 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:44:982.501 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830844981719 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:20:44:982.507 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:20:44:982.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:44:982.525 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748830844981748 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:20:44:982.531 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:20:45:287.982 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:288.069 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830845287800 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 288 dequeues = 288 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 337 dequeues = 337 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830817296822 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830844982513 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:20:45:288.083 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:45:288.091 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:288.321 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830845287933 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 40 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 51, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 288, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:20:45:288.346 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:45:296.435 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:296.537 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830845296400 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.28125 five_min_avg = 0.4525 fifteen_min_avg = 0.5025 ] cpu_share = 0.247917 ] ] 2025-06-02 Mon 02:20:45:296.553 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:20:45:296.565 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:296.586 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830845296416 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.247917 used_sys_mem = 4269467568 one_min_cpuload = 0.28125 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:20:45:296.596 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:20:45:296.608 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:296.831 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830845296448 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 623 in_bytes = 45856 out_tpkts = 741 out_bytes = 66372 exception_packets = 52 exception_packets_allowed = 37 raw_vhost_stats= [ name = vhost0 in_pkts = 623 in_bytes = 45856 out_pkts = 741 out_bytes = 66372 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 541 in_bytes = 57972 out_pkts = 623 out_bytes = 45194 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:20:45:296.856 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:20:45:296.865 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:20:45:296.893 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830845296482 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:20:45:296.899 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:20:47:296.971 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:20:47:297.214 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:20:47:297.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:47:297.349 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830847297053 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:20:47:297.357 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:20:47:297.365 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:20:47:297.595 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830847297309 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 82 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 40 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 51, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 294, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:20:47:297.620 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:20:59:334.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:35674 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:21:14:981.893 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:14:981.948 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748830874981856 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:21:14:981.960 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:21:14:982.088 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:14:982.229 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830874982060 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 630 in_bytes = 46330 out_pkts = 750 out_bytes = 66850 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 126 out_bw_usage = 127 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:21:14:982.270 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:21:14:982.286 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:14:982.302 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830874982065 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:21:14:982.311 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:21:15:288.328 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:15:288.536 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830875288125 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 299 dequeues = 299 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 349 dequeues = 349 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830847297060 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830874982284 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:21:15:288.581 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:21:15:288.599 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:15:289.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830875288268 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 83 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 299, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:21:15:289.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:21:15:296.726 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:15:296.953 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830875296676 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 630 in_bytes = 46330 out_tpkts = 750 out_bytes = 66850 raw_vhost_stats= [ name = vhost0 in_pkts = 630 in_bytes = 46330 out_pkts = 750 out_bytes = 66850 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 1.84856e-15 total_out_bandwidth_utilization = 1.51788e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 341, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 280, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 547 in_bytes = 58324 out_pkts = 630 out_bytes = 45668 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:21:15:296.983 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:21:15:296.993 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:15:297.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830875296735 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:21:15:297.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:21:17:297.365 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:21:17:297.432 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:21:17:297.441 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:21:17:297.545 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Idle 2025-06-02 Mon 02:21:17:297.628 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:17:297.665 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830877297471 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:21:17:297.673 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:21:17:297.689 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:21:17:297.719 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:17:297.951 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830877297618 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 85 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 303, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:21:17:297.972 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:21:44:982.373 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:44:982.440 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830904982172 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:21:44:982.486 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:21:44:982.496 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:44:982.564 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830904982482 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 748 in_bytes = 56966 out_pkts = 869 out_bytes = 84764 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 2836 out_bw_usage = 4777 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:21:44:982.590 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:21:44:982.602 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:44:982.620 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830904982488 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:21:44:982.628 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:21:45:288.535 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:288.615 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830905288381 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 308 dequeues = 308 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 360 dequeues = 360 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830877297544 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830904982600 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:21:45:288.629 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:21:45:288.639 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:288.906 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830905288509 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 86 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 308, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:21:45:288.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:21:45:296.923 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:297.002 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830905296890 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.83875 five_min_avg = 0.56375 fifteen_min_avg = 0.5375 ] cpu_share = 0.25625 ] ] 2025-06-02 Mon 02:21:45:297.012 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:21:45:297.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:297.037 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830905296905 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.25625 used_sys_mem = 4269467568 one_min_cpuload = 0.83875 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:21:45:297.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:21:45:297.051 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:297.217 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830905296943 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 748 in_bytes = 56966 out_tpkts = 869 out_bytes = 84764 exception_packets = 54 exception_packets_allowed = 39 raw_vhost_stats= [ name = vhost0 in_pkts = 748 in_bytes = 56966 out_pkts = 869 out_bytes = 84764 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 663 in_bytes = 76112 out_pkts = 748 out_bytes = 56304 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:21:45:297.238 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:21:45:297.246 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:21:45:297.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830905296975 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:21:45:297.279 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:21:46:804.794 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36780 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:21:46:849.555 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36780 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:21:47:297.679 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:21:47:297.951 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:21:47:298.057 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:47:298.085 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830907297803 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:21:47:298.095 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:21:47:298.106 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:21:47:298.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830907298045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 88 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 43 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 54, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 314, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:21:47:298.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:00:306.984 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:43866 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:22:14:982.489 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:14:982.646 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830934982451 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:22:14:982.662 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:22:14:982.673 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:14:982.781 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830934982635 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 761 in_bytes = 57906 out_pkts = 884 out_bytes = 85981 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 250 out_bw_usage = 324 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:22:14:982.797 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:22:14:982.807 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:14:982.819 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830934982727 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:22:14:982.826 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:22:15:288.841 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:15:288.953 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830935288672 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 319 dequeues = 319 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 372 dequeues = 372 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830907297810 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830934982805 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:22:15:288.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:15:288.990 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:15:289.227 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830935288796 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 89 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 319, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:22:15:289.247 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:15:297.232 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:15:297.530 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830935297193 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 761 in_bytes = 57906 out_tpkts = 884 out_bytes = 85981 exception_packets = 55 exception_packets_allowed = 40 raw_vhost_stats= [ name = vhost0 in_pkts = 761 in_bytes = 57906 out_pkts = 884 out_bytes = 85981 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 1.36447e-14 total_out_bandwidth_utilization = 8.36462e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 2517, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 1543, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 675 in_bytes = 77203 out_pkts = 761 out_bytes = 57244 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:22:15:297.565 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:22:15:297.599 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:15:297.642 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830935297237 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:22:15:297.650 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:22:17:298.078 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:22:17:298.159 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:22:17:298.228 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:22:17:298.349 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Idle 2025-06-02 Mon 02:22:17:298.443 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:17:298.483 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830937298286 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:22:17:298.492 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:22:17:298.500 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:22:17:298.528 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:17:298.763 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830937298431 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 91 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 323, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:22:17:298.786 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:44:982.430 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:44:982.487 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830964982391 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:22:44:982.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:22:44:982.585 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:44:982.657 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830964982559 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 767 in_bytes = 58338 out_pkts = 891 out_bytes = 86375 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 119 out_bw_usage = 108 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:22:44:982.734 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:22:44:982.749 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:44:982.760 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830964982579 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:22:44:982.767 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:22:45:289.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:289.183 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830965288892 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 328 dequeues = 328 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 383 dequeues = 383 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830937298347 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748830964982747 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:22:45:289.305 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:45:289.318 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:289.539 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748830965289035 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 92 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 328, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:22:45:289.568 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:22:45:298.001 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:298.101 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830965297935 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.49 five_min_avg = 0.51375 fifteen_min_avg = 0.52125 ] cpu_share = 0.25 ] ] 2025-06-02 Mon 02:22:45:298.114 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:22:45:298.125 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:298.145 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830965297971 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.25 used_sys_mem = 4269467568 one_min_cpuload = 0.49 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:22:45:298.153 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:22:45:298.162 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:298.383 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830965298016 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 767 in_bytes = 58338 out_tpkts = 891 out_bytes = 86375 raw_vhost_stats= [ name = vhost0 in_pkts = 767 in_bytes = 58338 out_pkts = 891 out_bytes = 86375 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 679 in_bytes = 77471 out_pkts = 767 out_bytes = 57676 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:22:45:298.415 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:22:45:298.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:22:45:298.534 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830965298048 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:22:45:298.545 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:22:47:298.495 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:22:47:299.009 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:22:47:299.152 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:47:299.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830967298574 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:22:47:299.188 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:22:47:299.196 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:22:47:299.562 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748830967299137 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 94 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 334, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:22:47:299.587 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:01:328.974 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:44468 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:23:14:984.766 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:14:985.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830994984728 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:23:14:985.499 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:23:14:985.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:14:985.622 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830994985017 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 774 in_bytes = 58812 out_pkts = 899 out_bytes = 86783 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 126 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:23:14:985.654 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:23:14:985.666 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:14:985.681 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748830994985025 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:23:14:985.690 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:23:15:289.342 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:15:289.452 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830995289227 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 339 dequeues = 339 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 395 dequeues = 395 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748830967298584 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748830994985664 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:23:15:289.467 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:15:289.490 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:15:289.711 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830995289302 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 95 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 339, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:23:15:289.730 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:15:298.241 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:15:298.464 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830995298209 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 774 in_bytes = 58812 out_tpkts = 899 out_bytes = 86783 raw_vhost_stats= [ name = vhost0 in_pkts = 774 in_bytes = 58812 out_pkts = 899 out_bytes = 86783 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 3.95734e-16 total_out_bandwidth_utilization = 6.50521e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 73, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 120, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 684 in_bytes = 77753 out_pkts = 774 out_bytes = 58150 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:23:15:298.493 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:23:15:298.504 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:15:298.532 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748830995298256 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:23:15:298.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:23:17:299.221 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:23:17:299.307 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:23:17:299.351 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:23:17:299.452 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:23:17:299.565 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:17:299.597 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830997299392 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:23:17:299.759 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:23:17:299.772 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:23:17:299.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:17:300.074 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748830997299553 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 97 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 343, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:23:17:300.095 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:44:984.654 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:44:984.789 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831024984619 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:23:44:984.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:23:44:984.917 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:44:985.012 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831024984894 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 783 in_bytes = 59466 out_pkts = 909 out_bytes = 87399 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 180 out_bw_usage = 169 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:23:44:985.035 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:23:44:985.044 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:44:985.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831024984906 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:23:44:985.061 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:23:45:289.555 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:290.432 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831025289404 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 348 dequeues = 348 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 406 dequeues = 406 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748830997299450 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831024985042 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:23:45:290.461 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:45:290.476 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:290.721 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831025289530 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 98 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 348, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:23:45:290.749 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:23:45:298.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:298.587 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831025298420 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.45125 five_min_avg = 0.48875 fifteen_min_avg = 0.51125 ] cpu_share = 0.254167 ] ] 2025-06-02 Mon 02:23:45:298.605 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:23:45:298.620 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:298.644 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831025298435 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.254167 used_sys_mem = 4269467568 one_min_cpuload = 0.45125 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:23:45:298.653 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:23:45:298.664 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:298.945 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831025298462 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 783 in_bytes = 59466 out_tpkts = 909 out_bytes = 87399 exception_packets = 56 exception_packets_dropped = 16 raw_vhost_stats= [ name = vhost0 in_pkts = 783 in_bytes = 59466 out_pkts = 909 out_bytes = 87399 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 691 in_bytes = 78243 out_pkts = 783 out_bytes = 58804 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:23:45:298.982 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:23:45:299.004 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:23:45:299.052 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831025298487 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:23:45:299.062 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:23:47:299.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:23:47:299.878 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:23:47:299.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:47:300.007 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831027299708 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:23:47:300.025 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:23:47:300.033 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:23:47:300.250 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831027299968 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 354, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:23:47:300.269 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:02:314.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:56076 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:24:14:984.761 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:14:984.883 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831054984691 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:24:14:984.895 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:24:14:984.920 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:14:984.987 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831054984878 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 794 in_bytes = 60252 out_pkts = 921 out_bytes = 88119 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 216 out_bw_usage = 198 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:24:14:985.018 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:24:14:985.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:14:985.038 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831054984910 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:24:14:985.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:24:15:289.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:15:290.035 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831055289730 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 359 dequeues = 359 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 418 dequeues = 418 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831027299718 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831054985026 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:24:15:290.052 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:15:290.064 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:15:290.391 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831055289831 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 101 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 359, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:24:15:290.449 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:15:298.858 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:15:299.066 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831055298816 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 794 in_bytes = 60252 out_tpkts = 921 out_bytes = 88119 exception_packets = 58 exception_packets_allowed = 42 raw_vhost_stats= [ name = vhost0 in_pkts = 794 in_bytes = 60252 out_pkts = 921 out_bytes = 88119 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 635 out_bandwidth_usage = 457 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] total_in_bandwidth_utilization = 7.80626e-16 total_out_bandwidth_utilization = 1.06794e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 144, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 197, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 700 in_bytes = 78837 out_pkts = 795 out_bytes = 59632 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:24:15:299.097 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:24:15:299.111 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:15:299.139 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831055298855 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:24:15:299.146 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:24:17:300.070 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:24:17:300.179 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:24:17:300.233 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:24:17:300.349 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Idle 2025-06-02 Mon 02:24:17:300.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:17:300.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831057300275 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:24:17:300.536 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:24:17:300.544 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:24:17:300.581 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:17:300.927 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831057300450 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 103 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 363, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:24:17:300.956 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:44:984.997 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:44:985.057 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831084984935 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:24:44:985.068 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:24:44:985.115 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:44:985.331 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831084985091 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 802 in_bytes = 60816 out_pkts = 929 out_bytes = 88575 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 150 out_bw_usage = 121 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:24:44:985.361 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:24:44:985.371 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:44:985.384 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831084985102 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:24:44:985.393 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:24:45:290.445 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:290.588 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831085290229 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 368 dequeues = 368 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 429 dequeues = 429 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748831057300347 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831084985369 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:24:45:290.602 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:45:290.612 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:290.886 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831085290330 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 104 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 368, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:24:45:290.916 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:24:45:299.089 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:299.237 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831085299053 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.27875 five_min_avg = 0.43 fifteen_min_avg = 0.49 ] cpu_share = 0.26875 ] ] 2025-06-02 Mon 02:24:45:299.249 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:24:45:299.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:299.274 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831085299074 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.26875 used_sys_mem = 4269467568 one_min_cpuload = 0.27875 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:24:45:299.281 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:24:45:299.288 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:299.411 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831085299106 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 802 in_bytes = 60816 out_tpkts = 929 out_bytes = 88575 raw_vhost_stats= [ name = vhost0 in_pkts = 802 in_bytes = 60816 out_pkts = 929 out_bytes = 88575 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 19 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 705 in_bytes = 79167 out_pkts = 803 out_bytes = 60196 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 14 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 14 ], ] ] ] 2025-06-02 Mon 02:24:45:299.425 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:24:45:299.436 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:24:45:299.463 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831085299133 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:24:45:299.469 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:24:47:300.538 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:24:47:300.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:24:47:301.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:47:301.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831087300692 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:24:47:301.259 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:24:47:301.271 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:24:47:301.716 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831087301153 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 106 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 374, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:24:47:301.756 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:03:302.894 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:49502 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:25:14:984.793 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:14:984.868 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831114984740 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:25:14:984.881 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:25:14:984.935 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:14:985.027 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831114984917 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 812 in_bytes = 61464 out_pkts = 939 out_bytes = 89115 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 178 out_bw_usage = 148 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 237 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 242 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:25:14:985.053 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:25:14:985.063 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:14:985.075 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831114984922 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:25:14:985.081 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:25:15:290.630 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:15:290.752 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831115290491 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 379 dequeues = 379 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 441 dequeues = 441 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831087300705 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831114985062 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:25:15:290.766 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:15:290.873 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:15:291.283 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831115290584 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 107 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 379, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:25:15:291.336 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:15:299.335 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:15:299.586 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831115299299 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 812 in_bytes = 61464 out_tpkts = 939 out_bytes = 89115 exception_packets = 59 exception_packets_allowed = 43 raw_vhost_stats= [ name = vhost0 in_pkts = 812 in_bytes = 61464 out_pkts = 939 out_bytes = 89115 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 237 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 242 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 237 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 242 ] total_in_bandwidth_utilization = 5.3668e-16 total_out_bandwidth_utilization = 8.72783e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 99, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 161, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 712 in_bytes = 79581 out_pkts = 813 out_bytes = 60844 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 237 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 237 ], ] ] ] 2025-06-02 Mon 02:25:15:299.613 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:25:15:299.626 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:15:299.661 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831115299345 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:25:15:299.669 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:25:17:301.127 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:25:17:301.191 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:25:17:301.202 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:25:17:301.322 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Idle 2025-06-02 Mon 02:25:17:301.420 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:17:301.455 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831117301228 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:25:17:301.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:25:17:301.488 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:25:17:301.523 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:17:301.859 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831117301406 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 109 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 383, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:25:17:301.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:44:985.600 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:44:985.660 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831144985489 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:25:44:985.674 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:25:44:985.758 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:44:985.865 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831144985732 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 870 in_bytes = 66591 out_pkts = 1025 out_bytes = 98718 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 1367 out_bw_usage = 2560 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:25:44:985.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:25:44:985.905 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:44:985.922 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831144985748 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:25:44:985.931 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:25:45:290.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:290.902 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831145290676 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 388 dequeues = 388 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 452 dequeues = 452 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748831117301321 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831144985903 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:25:45:290.915 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:45:290.923 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:291.267 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831145290798 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 110 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 388, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:25:45:291.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:25:45:299.693 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:299.802 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831145299605 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.61375 five_min_avg = 0.47375 fifteen_min_avg = 0.5 ] cpu_share = 0.28125 ] ] 2025-06-02 Mon 02:25:45:299.826 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:25:45:299.835 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:299.850 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831145299622 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.28125 used_sys_mem = 4269467568 one_min_cpuload = 0.61375 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:25:45:299.857 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:25:45:299.863 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:300.001 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831145299650 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 870 in_bytes = 66591 out_tpkts = 1025 out_bytes = 98718 exception_packets = 60 exception_packets_allowed = 44 raw_vhost_stats= [ name = vhost0 in_pkts = 870 in_bytes = 66591 out_pkts = 1025 out_bytes = 98718 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 795 in_bytes = 89058 out_pkts = 871 out_bytes = 65971 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:25:45:300.014 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:25:45:300.021 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:25:45:300.048 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831145299675 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:25:45:300.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:25:47:301.448 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:25:47:301.634 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:25:47:301.742 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:47:301.768 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831147301510 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:25:47:301.776 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:25:47:301.783 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:25:47:302.710 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831147301730 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 112 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 394, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:25:47:302.763 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:04:274.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:49954 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:26:09:057.715 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:49970 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:26:09:101.897 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:49970 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:26:14:985.710 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:14:985.780 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831174985673 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:26:14:985.790 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:26:14:985.905 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:14:986.005 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831174985886 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 942 in_bytes = 72966 out_pkts = 1106 out_bytes = 109580 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 1700 out_bw_usage = 2896 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:26:14:986.034 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:26:14:986.043 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:14:986.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831174985896 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:26:14:986.060 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:26:15:291.145 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:15:291.253 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831175290986 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 399 dequeues = 399 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 464 dequeues = 464 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831147301516 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831174986042 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:26:15:291.265 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:15:291.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:15:291.512 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831175291116 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 113 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 50 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 61, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 399, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:26:15:291.540 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:15:299.938 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:15:300.289 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831175299900 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 942 in_bytes = 72966 out_tpkts = 1106 out_bytes = 109580 exception_packets = 61 exception_packets_allowed = 45 raw_vhost_stats= [ name = vhost0 in_pkts = 942 in_bytes = 72966 out_pkts = 1106 out_bytes = 109580 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] total_in_bandwidth_utilization = 1.46096e-14 total_out_bandwidth_utilization = 8.31041e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 2695, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 1533, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 873 in_bytes = 99794 out_pkts = 943 out_bytes = 72346 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:26:15:300.329 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:26:15:300.345 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:15:300.395 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831175299946 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:26:15:300.406 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:26:17:301.751 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:26:17:302.542 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:26:17:302.559 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:26:17:302.831 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:26:17:302.937 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:17:302.963 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831177302589 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:26:17:302.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:26:17:302.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:26:17:303.019 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:17:303.733 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831177302925 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 115 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 50 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 61, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 403, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:26:17:303.798 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:19:361.131 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:44404 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:26:19:409.664 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:44404 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:26:29:895.832 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:39472 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:26:29:937.785 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:39472 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:26:44:987.299 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:44:987.389 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831204987258 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:26:44:987.428 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:26:44:987.583 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:44:987.743 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831204987542 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 958 in_bytes = 74150 out_pkts = 1123 out_bytes = 111188 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 315 out_bw_usage = 428 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:26:44:987.774 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:26:44:987.787 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:44:987.799 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831204987563 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:26:44:987.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:26:45:291.564 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:291.643 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831205291228 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 408 dequeues = 408 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 475 dequeues = 475 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748831177302830 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831204987786 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:26:45:291.655 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:45:291.666 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:291.911 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831205291422 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 116 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 65, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 408, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:26:45:291.947 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:26:45:300.205 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:300.278 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831205300171 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.5375 five_min_avg = 0.48875 fifteen_min_avg = 0.50375 ] cpu_share = 0.245833 ] ] 2025-06-02 Mon 02:26:45:300.288 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:26:45:300.297 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:300.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831205300194 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.245833 used_sys_mem = 4269467568 one_min_cpuload = 0.5375 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:26:45:300.319 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:26:45:300.326 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:300.468 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831205300228 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 958 in_bytes = 74150 out_tpkts = 1123 out_bytes = 111188 exception_packets = 62 exception_packets_allowed = 46 raw_vhost_stats= [ name = vhost0 in_pkts = 958 in_bytes = 74150 out_pkts = 1123 out_bytes = 111188 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 887 in_bytes = 101276 out_pkts = 959 out_bytes = 73530 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:26:45:300.495 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:26:45:300.503 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:26:45:300.530 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831205300262 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:26:45:300.536 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:26:47:302.985 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:26:47:303.221 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:26:47:303.361 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:47:303.390 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831207303063 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:26:47:303.397 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:26:47:303.405 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:26:47:303.699 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831207303347 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 118 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 65, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 414, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:26:47:303.760 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:05:295.231 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54638 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:27:14:989.373 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:14:989.467 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831234989338 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:27:14:989.482 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:27:14:989.493 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:14:989.567 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831234989471 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 965 in_bytes = 74624 out_pkts = 1130 out_bytes = 111554 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 126 out_bw_usage = 97 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:27:14:989.598 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:27:14:989.613 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:14:989.626 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831234989479 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:27:14:989.633 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:27:15:291.781 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:15:291.929 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831235291596 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 419 dequeues = 419 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 487 dequeues = 487 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831207303069 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831234989611 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:27:15:291.963 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:15:291.977 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:15:292.196 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831235291744 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 119 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 66, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 419, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:27:15:292.224 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:15:300.465 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:15:300.930 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831235300431 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 965 in_bytes = 74624 out_tpkts = 1130 out_bytes = 111554 raw_vhost_stats= [ name = vhost0 in_pkts = 965 in_bytes = 74624 out_pkts = 1130 out_bytes = 111554 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] total_in_bandwidth_utilization = 1.24141e-15 total_out_bandwidth_utilization = 1.19804e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 229, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 221, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 891 in_bytes = 101516 out_pkts = 966 out_bytes = 74004 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:27:15:301.026 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:27:15:301.041 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:15:301.070 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831235300468 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:27:15:301.076 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:27:17:303.382 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:27:17:303.466 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:27:17:303.505 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:27:17:303.621 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Idle 2025-06-02 Mon 02:27:17:303.749 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:17:303.779 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831237303541 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:27:17:303.797 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:27:17:303.805 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:27:17:303.839 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:17:304.086 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831237303721 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 121 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 66, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 423, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:27:17:304.110 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:34:182.480 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54678 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-06-02 Mon 02:27:34:229.677 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:54678 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2025-06-02 Mon 02:27:44:990.915 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:44:990.981 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831264990877 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:27:44:990.997 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:27:44:991.085 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:44:991.158 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831264991051 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 977 in_bytes = 75474 out_pkts = 1142 out_bytes = 112541 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 226 out_bw_usage = 263 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:27:44:991.180 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:27:44:991.188 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:44:991.198 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831264991077 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:27:44:991.204 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:27:45:292.112 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:292.233 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831265291895 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 428 dequeues = 428 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 498 dequeues = 498 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748831237303618 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831264991187 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:27:45:292.254 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:45:292.267 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:292.678 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831265292060 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 57 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 68, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 428, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:27:45:292.725 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:27:45:300.676 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:300.794 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831265300634 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.45625 five_min_avg = 0.46625 fifteen_min_avg = 0.49375 ] cpu_share = 0.252083 ] ] 2025-06-02 Mon 02:27:45:300.809 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:27:45:300.822 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:300.843 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831265300651 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.252083 used_sys_mem = 4269467568 one_min_cpuload = 0.45625 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:27:45:300.855 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:27:45:300.864 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:301.028 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831265300678 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 977 in_bytes = 75474 out_tpkts = 1142 out_bytes = 112541 exception_packets = 63 exception_packets_allowed = 47 raw_vhost_stats= [ name = vhost0 in_pkts = 977 in_bytes = 75474 out_pkts = 1142 out_bytes = 112541 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 900 in_bytes = 102377 out_pkts = 978 out_bytes = 74854 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:27:45:301.054 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:27:45:301.062 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:27:45:301.105 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831265300710 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:27:45:301.112 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:27:47:303.745 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:27:47:303.972 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:27:47:304.084 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:47:304.111 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831267303838 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:27:47:304.119 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:27:47:304.129 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:27:47:304.341 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831267304071 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 57 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 68, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 434, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:27:47:304.362 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:06:279.871 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:36576 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:28:14:992.565 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:14:992.637 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831294992526 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:28:14:992.652 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:28:14:993.061 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:14:993.154 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831294993033 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 986 in_bytes = 76080 out_pkts = 1151 out_bytes = 113039 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 161 out_bw_usage = 132 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:28:14:993.181 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:28:14:993.190 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:14:993.201 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831294993038 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:28:14:993.208 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:28:15:292.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:15:292.412 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831295292204 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 439 dequeues = 439 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 510 dequeues = 510 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831267303845 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831294993189 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:28:15:292.441 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:15:292.450 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:15:292.665 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831295292285 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 125 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 439, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:28:15:292.682 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:15:300.989 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:15:301.205 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831295300945 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 986 in_bytes = 76080 out_tpkts = 1151 out_bytes = 113039 exception_packets = 64 exception_packets_allowed = 48 raw_vhost_stats= [ name = vhost0 in_pkts = 986 in_bytes = 76080 out_pkts = 1151 out_bytes = 113039 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] total_in_bandwidth_utilization = 8.89046e-16 total_out_bandwidth_utilization = 1.05168e-15 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 164, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 194, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 906 in_bytes = 102749 out_pkts = 987 out_bytes = 75460 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:28:15:301.233 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:28:15:301.243 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:15:301.271 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831295301040 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:28:15:301.277 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:28:17:304.092 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:28:17:304.230 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:28:17:304.269 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:28:17:304.363 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Idle 2025-06-02 Mon 02:28:17:304.472 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:17:304.514 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831297304302 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:28:17:304.524 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:28:17:304.536 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:28:17:304.586 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:17:304.856 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831297304454 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 127 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 443, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:28:17:304.882 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:44:992.712 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:44:992.808 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831324992654 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:28:44:992.821 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:28:44:993.025 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:44:993.136 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831324993001 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 993 in_bytes = 76554 out_pkts = 1159 out_bytes = 113475 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 130 out_bw_usage = 120 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:28:44:993.168 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:28:44:993.181 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:44:993.197 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831324993010 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:28:44:993.207 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:28:45:292.707 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:292.821 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831325292544 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 448 dequeues = 448 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 521 dequeues = 521 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1748831297304361 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1748831324993179 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:28:45:292.836 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:45:292.844 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:293.153 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831325292665 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 128 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 448, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:28:45:293.190 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:28:45:301.510 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:301.601 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831325301462 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ num_cpu = 8 sys_mem_info= [ total = 25603328 used = 0 free = 4137221678 buffers = 32720 cached = 25467008 node_type = ] meminfo= [ virt = 32720 peakvirt = 4123187608 res = 32720 ] cpuload= [ one_min_avg = 0.30875 five_min_avg = 0.42375 fifteen_min_avg = 0.4775 ] cpu_share = 0.247917 ] ] 2025-06-02 Mon 02:28:45:301.613 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:28:45:301.623 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:301.638 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831325301489 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 cpu_info= [ [ [ mem_virt = 32720 cpu_share = 0.247917 used_sys_mem = 4269467568 one_min_cpuload = 0.30875 mem_res = 32720 ], ] ] ] 2025-06-02 Mon 02:28:45:301.645 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-06-02 Mon 02:28:45:301.651 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:302.044 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831325301516 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 993 in_bytes = 76554 out_tpkts = 1159 out_bytes = 113475 raw_vhost_stats= [ name = vhost0 in_pkts = 993 in_bytes = 76554 out_pkts = 1159 out_bytes = 113475 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 911 in_bytes = 103059 out_pkts = 994 out_bytes = 75934 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:28:45:302.082 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:28:45:302.096 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:28:45:302.124 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: SANDESH: Send FAILED: 1748831325301541 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:28:45:302.132 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535320090368, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:28:47:304.543 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-06-02 Mon 02:28:47:304.932 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Connect : Start Connect timer 10.0.0.129:8086 2025-06-02 Mon 02:28:47:305.071 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:47:305.104 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831327304651 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:28:47:305.126 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:28:47:305.136 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:28:47:305.434 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831327305050 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 130 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 454, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:28:47:305.462 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:29:07:207.540 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535498590528, Pid 1167645]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8085::10.0.0.129:53966 > SSL Handshake failed due to error: 336130315 category: asio.ssl message: wrong version number src/contrail-common/io/ssl_server.cc 96 2025-06-02 Mon 02:29:14:993.376 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:14:993.443 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: SANDESH: Send FAILED: 1748831354993335 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:29:14:993.455 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535341082368, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:29:14:993.611 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:14:993.774 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831354993584 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 1003 in_bytes = 77202 out_pkts = 1172 out_bytes = 114197 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 172 out_bw_usage = 192 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ] 2025-06-02 Mon 02:29:14:993.817 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: UVEVMIStats 2025-06-02 Mon 02:29:14:993.832 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:14:993.849 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: SANDESH: Send FAILED: 1748831354993592 [SYS_INFO]: EndpointSecurityStats: name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 eps= [ [ ] ] file = controller/src/vnsw/agent/uve/interface_uve_stats_table.cc line = 135 2025-06-02 Mon 02:29:14:993.858 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535345280768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: EndpointSecurityStats 2025-06-02 Mon 02:29:15:293.102 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:15:293.222 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831355293003 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 459 dequeues = 459 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 533 dequeues = 533 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1748831327304661 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1748831354993830 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 7 ] 2025-06-02 Mon 02:29:15:293.235 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:29:15:293.243 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:15:293.472 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831355293078 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 131 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 459, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:29:15:293.489 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:29:15:301.850 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:15:302.072 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831355301817 [SYS_NOTICE]: VrouterStats: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 in_tpkts = 1003 in_bytes = 77202 out_tpkts = 1172 out_bytes = 114197 exception_packets = 66 exception_packets_allowed = 50 raw_vhost_stats= [ name = vhost0 in_pkts = 1003 in_bytes = 77202 out_pkts = 1172 out_bytes = 114197 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 661 out_bandwidth_usage = 453 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] total_in_bandwidth_utilization = 6.50521e-16 total_out_bandwidth_utilization = 8.40257e-16 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 120, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 155, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 921 in_bytes = 103655 out_pkts = 1005 out_bytes = 76624 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 318 ], ] ] ] 2025-06-02 Mon 02:29:15:302.219 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterStats 2025-06-02 Mon 02:29:15:302.235 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Processing scm::EvSandeshSend in state Connect 2025-06-02 Mon 02:29:15:302.266 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: SANDESH: Send FAILED: 1748831355301852 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.129 [ reconnects = 1 in_msgs = 10 out_msgs = 31 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 3 inet4_multicast = 0 evpn = 2 bridge = 3 inet6_unicast = 3 ], ] ] ] 2025-06-02 Mon 02:29:15:302.273 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: Wrong state: Connect for event: EvSandeshSend message: VrouterControlStatsTrace 2025-06-02 Mon 02:29:17:305.065 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535349479168, Pid 1167645]: 10.0.0.129:8086 EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:29:17:305.146 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvConnectTimerExpired in state Connect 2025-06-02 Mon 02:29:17:305.183 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Connect : EvConnectTimerExpired 2025-06-02 Mon 02:29:17:305.281 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Idle 2025-06-02 Mon 02:29:17:305.376 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:29:17:305.401 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831357305219 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:5269, ] ] status = Up description = OpenConfirm ], [ type = XMPP name = dns-server:10.0.0.129 server_addrs= [ [ (*_iter6) = 10.0.0.129:53, ] ] status = Up description = OpenConfirm ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-06-02 Mon 02:29:17:305.418 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-06-02 Mon 02:29:17:305.426 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvTcpDeleteSession in state Idle 2025-06-02 Mon 02:29:17:305.451 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:29:17:305.679 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: SANDESH: Send FAILED: 1748831357305363 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-542-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1167645 http_port = 8085 start_time = 1748830125279824 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = EndpointSecurityStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 133 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMIStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UVEVMITags [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVMInterfaceAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VnObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 463, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-06-02 Mon 02:29:17:305.716 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535332685568, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-06-02 Mon 02:29:44:994.045 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:29:44:994.107 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: SANDESH: Send FAILED: 1748831384994010 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = default-domain:default-project:ip-fabric ingress_flow_count = 0 egress_flow_count = 0 ] 2025-06-02 Mon 02:29:44:994.122 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535324288768, Pid 1167645]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-06-02 Mon 02:29:44:994.312 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: Processing scm::EvSandeshSend in state Idle 2025-06-02 Mon 02:29:44:994.455 UTC cn-jenkins-deploy-platform-juju-os-542-1 [Thread 140535336883968, Pid 1167645]: SANDESH: Send FAILED: 1748831384994198 [SYS_NOTICE]: UVEVMIStats: data= [ name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-542-1:vhost0 vm_name = vm_uuid = virtual_network = default-domain:default-project:ip-fabric raw_if_stats= [ in_pkts = 1009 in_bytes = 77634 out_pkts = 1178 out_bytes = 114521 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] in_bw_usage = 115 out_bw_usage = 86 raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 318 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 5 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 323 ] policy_rules= [ [ ] ] ]