2025-09-01 Mon 01:35:47:493.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: vhost interface name : vhost0 2025-09-01 Mon 01:35:47:493.219 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: vhost IP Address : 10.0.0.22/24 2025-09-01 Mon 01:35:47:493.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: vhost gateway : 0.0.0.0 2025-09-01 Mon 01:35:47:493.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Ethernet ports : ens3 2025-09-01 Mon 01:35:47:493.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Loopback IP : 0.0.0.0 2025-09-01 Mon 01:35:47:493.238 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Xmpp Servers : 10.0.0.50:5269 2025-09-01 Mon 01:35:47:493.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Xmpp Authentication : 0 2025-09-01 Mon 01:35:47:493.248 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Cluster-Name : 2025-09-01 Mon 01:35:47:493.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: DNS Servers : 10.0.0.50:53 2025-09-01 Mon 01:35:47:493.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: DNS client port : 8098 2025-09-01 Mon 01:35:47:493.260 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: DNS timeout : 3000 2025-09-01 Mon 01:35:47:493.264 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: DNS max retries : 2 2025-09-01 Mon 01:35:47:493.268 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Xmpp Dns Authentication : 0 2025-09-01 Mon 01:35:47:493.272 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: TSN Servers : [] 2025-09-01 Mon 01:35:47:493.276 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: COLLECTOR Servers : 10.0.0.50:8086 2025-09-01 Mon 01:35:47:493.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Tunnel-Type : MPLSoGRE 2025-09-01 Mon 01:35:47:493.283 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Metadata-Proxy Shared Secret: contrail 2025-09-01 Mon 01:35:47:493.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Metadata-Proxy Port : 8097 2025-09-01 Mon 01:35:47:493.291 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Metadata-Proxy SSL Flag : 0 2025-09-01 Mon 01:35:47:493.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Linklocal Max System Flows : 16384 2025-09-01 Mon 01:35:47:493.298 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Linklocal Max Vm Flows : 16384 2025-09-01 Mon 01:35:47:493.302 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow cache timeout : 0 2025-09-01 Mon 01:35:47:493.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Stale Interface cleanup timeout : 60 2025-09-01 Mon 01:35:47:493.310 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow thread count : 1 2025-09-01 Mon 01:35:47:493.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow latency limit : 0 2025-09-01 Mon 01:35:47:493.318 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow index-mgr sm log count : 0 2025-09-01 Mon 01:35:47:493.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow add-tokens : 50 2025-09-01 Mon 01:35:47:493.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow ksync-tokens : 25 2025-09-01 Mon 01:35:47:493.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow del-tokens : 16 2025-09-01 Mon 01:35:47:493.336 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow update-tokens : 16 2025-09-01 Mon 01:35:47:493.340 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Pin flow netlink task to CPU: 2025-09-01 Mon 01:35:47:493.344 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Maximum sessions : 100 2025-09-01 Mon 01:35:47:493.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Maximum session aggregates : 8 2025-09-01 Mon 01:35:47:493.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Maximum session endpoints : 5 2025-09-01 Mon 01:35:47:493.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Fabric SNAT hash table size : 4096 2025-09-01 Mon 01:35:47:493.360 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Flow excluding Router ID in hash :0 2025-09-01 Mon 01:35:47:493.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Agent Mode : Vrouter 2025-09-01 Mon 01:35:47:493.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Gateway Mode : None 2025-09-01 Mon 01:35:47:493.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: DHCP Relay Mode : 0 2025-09-01 Mon 01:35:47:493.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance netns cmd : /usr/local/bin/opencontrail-vrouter-netns 2025-09-01 Mon 01:35:47:493.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance docker cmd : /usr/local/bin/opencontrail-vrouter-docker 2025-09-01 Mon 01:35:47:493.385 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance workers : 0 2025-09-01 Mon 01:35:47:493.389 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance timeout : 0 2025-09-01 Mon 01:35:47:493.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance lb ssl : 2025-09-01 Mon 01:35:47:493.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Service instance lbaas auth : 2025-09-01 Mon 01:35:47:493.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Bgp as a service port range : 50000-50512 2025-09-01 Mon 01:35:47:493.404 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Services queue limit : 1024 2025-09-01 Mon 01:35:47:493.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: BGPAAS max shared sessions for service port : 4 2025-09-01 Mon 01:35:47:493.411 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Sandesh Key file : /etc/contrail/ssl/private/server-privkey.pem 2025-09-01 Mon 01:35:47:493.415 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Sandesh Cert file : /etc/contrail/ssl/certs/server.pem 2025-09-01 Mon 01:35:47:493.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Sandesh CA Cert : /etc/contrail/ssl/certs/ca-cert.pem 2025-09-01 Mon 01:35:47:493.423 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Sandesh SSL Enable : 0 2025-09-01 Mon 01:35:47:493.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Introspect SSL Enable : 0 2025-09-01 Mon 01:35:47:493.433 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Hypervisor mode : kvm 2025-09-01 Mon 01:35:47:493.507 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Ethernet Port Encap Type : 2025-09-01 Mon 01:35:47:493.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: Platform mode : Vrouter on host linux kernel 2025-09-01 Mon 01:35:47:493.731 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: VRouter [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"v0+nightly","build-time":"2025-09-01 00:13:09.128962","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10717-1.novalocal","build-id":"unknown","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-09-01 Mon 01:35:47:493.758 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Logging: DISABLED -> ENABLED 2025-09-01 Mon 01:35:47:493.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-09-01 Mon 01:35:47:507.749 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.787 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> self 2025-09-01 Mon 01:35:47:507.793 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:35:47:507.798 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:35:47:507.803 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-access-control-list -> self 2025-09-01 Mon 01:35:47:507.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap access-control-list : self -> virtual-network-access-control-list 2025-09-01 Mon 01:35:47:507.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap access-control-list : virtual-network-access-control-list -> nil 2025-09-01 Mon 01:35:47:507.820 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.823 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> self 2025-09-01 Mon 01:35:47:507.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:35:47:507.832 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> virtual-network-network-ipam 2025-09-01 Mon 01:35:47:507.836 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:35:47:507.840 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.844 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-qos-config -> self 2025-09-01 Mon 01:35:47:507.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-config : self -> virtual-network-qos-config 2025-09-01 Mon 01:35:47:507.862 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-config : virtual-network-qos-config -> nil 2025-09-01 Mon 01:35:47:507.866 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.870 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-security-logging-object -> self 2025-09-01 Mon 01:35:47:507.874 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : self -> virtual-network-security-logging-object 2025-09-01 Mon 01:35:47:507.878 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : virtual-network-security-logging-object -> nil 2025-09-01 Mon 01:35:47:507.882 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.886 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : logical-router-virtual-network -> self 2025-09-01 Mon 01:35:47:507.890 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router-virtual-network : self -> logical-router-virtual-network 2025-09-01 Mon 01:35:47:507.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router-virtual-network : logical-router-virtual-network -> logical-router-virtual-network 2025-09-01 Mon 01:35:47:507.899 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router : self -> logical-router-virtual-network 2025-09-01 Mon 01:35:47:507.903 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router : logical-router-virtual-network -> nil 2025-09-01 Mon 01:35:47:507.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:35:47:507.911 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-multicast-policy -> self 2025-09-01 Mon 01:35:47:507.915 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap multicast-policy : self -> virtual-network-multicast-policy 2025-09-01 Mon 01:35:47:507.919 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap multicast-policy : virtual-network-multicast-policy -> nil 2025-09-01 Mon 01:35:47:507.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for routing-instance 2025-09-01 Mon 01:35:47:507.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> self 2025-09-01 Mon 01:35:47:507.938 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-routing-instance 2025-09-01 Mon 01:35:47:507.942 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-routing-instance 2025-09-01 Mon 01:35:47:507.946 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:35:47:507.952 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:35:47:507.957 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for security-group 2025-09-01 Mon 01:35:47:507.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-group : security-group-access-control-list -> self 2025-09-01 Mon 01:35:47:507.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap access-control-list : self -> security-group-access-control-list 2025-09-01 Mon 01:35:47:507.969 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap access-control-list : security-group-access-control-list -> nil 2025-09-01 Mon 01:35:47:507.974 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for tag 2025-09-01 Mon 01:35:47:507.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> self 2025-09-01 Mon 01:35:47:507.982 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:507.994 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : policy-management-application-policy-set -> application-policy-set-tag 2025-09-01 Mon 01:35:47:507.999 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap policy-management : policy-management-application-policy-set -> nil 2025-09-01 Mon 01:35:47:508.004 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.008 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:35:47:508.012 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:35:47:508.024 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:35:47:508.028 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.032 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-09-01 Mon 01:35:47:508.037 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:35:47:508.042 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : virtual-machine-interface-virtual-machine -> nil 2025-09-01 Mon 01:35:47:508.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.058 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-09-01 Mon 01:35:47:508.064 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:35:47:508.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> nil 2025-09-01 Mon 01:35:47:508.076 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.081 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:35:47:508.087 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.092 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.096 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:35:47:508.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:35:47:508.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-routing-instance -> self 2025-09-01 Mon 01:35:47:508.113 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-routing-instance : self -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:35:47:508.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-routing-instance : virtual-machine-interface-routing-instance -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:35:47:508.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : self -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:35:47:508.125 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : virtual-machine-interface-routing-instance -> nil 2025-09-01 Mon 01:35:47:508.129 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.133 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-group -> self 2025-09-01 Mon 01:35:47:508.137 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-group : self -> virtual-machine-interface-security-group 2025-09-01 Mon 01:35:47:508.141 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-group : virtual-machine-interface-security-group -> nil 2025-09-01 Mon 01:35:47:508.148 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.152 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : alias-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.156 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap alias-ip : self -> alias-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.160 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap alias-ip : alias-ip-pool-alias-ip -> alias-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.164 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap alias-ip-pool : virtual-network-alias-ip-pool -> alias-ip-pool-alias-ip 2025-09-01 Mon 01:35:47:508.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-alias-ip-pool 2025-09-01 Mon 01:35:47:508.180 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-alias-ip-pool -> nil 2025-09-01 Mon 01:35:47:508.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.189 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.202 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-09-01 Mon 01:35:47:508.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:35:47:508.210 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:35:47:508.214 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:35:47:508.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:35:47:508.222 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-09-01 Mon 01:35:47:508.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:35:47:508.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:35:47:508.251 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:35:47:508.255 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:35:47:508.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.263 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.266 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.270 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap floating-ip : instance-ip-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.274 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap instance-ip : instance-ip-virtual-network -> instance-ip-floating-ip 2025-09-01 Mon 01:35:47:508.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> instance-ip-virtual-network 2025-09-01 Mon 01:35:47:508.282 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : instance-ip-virtual-network -> nil 2025-09-01 Mon 01:35:47:508.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : instance-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.294 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap instance-ip : self -> instance-ip-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.298 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap instance-ip : instance-ip-virtual-machine-interface -> nil 2025-09-01 Mon 01:35:47:508.302 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-route-table -> self 2025-09-01 Mon 01:35:47:508.310 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap interface-route-table : self -> virtual-machine-interface-route-table 2025-09-01 Mon 01:35:47:508.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap interface-route-table : virtual-machine-interface-route-table -> nil 2025-09-01 Mon 01:35:47:508.318 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.324 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : subnet-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.329 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap subnet : self -> subnet-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap subnet : subnet-virtual-machine-interface -> nil 2025-09-01 Mon 01:35:47:508.337 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.341 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.346 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> logical-interface-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-09-01 Mon 01:35:47:508.354 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:35:47:508.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.366 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-port-group-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> virtual-port-group-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.375 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : physical-router-physical-interface -> virtual-port-group-physical-interface 2025-09-01 Mon 01:35:47:508.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:35:47:508.382 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.391 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.398 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-as-a-service : bgpaas-bgp-router -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-router : self -> bgpaas-bgp-router 2025-09-01 Mon 01:35:47:508.409 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-router : instance-bgp-router -> bgpaas-bgp-router 2025-09-01 Mon 01:35:47:508.413 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : self -> instance-bgp-router 2025-09-01 Mon 01:35:47:508.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : instance-bgp-router -> nil 2025-09-01 Mon 01:35:47:508.421 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.425 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.433 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-as-a-service : bgpaas-health-check -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.437 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-health-check : self -> bgpaas-health-check 2025-09-01 Mon 01:35:47:508.441 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-health-check : bgpaas-health-check -> nil 2025-09-01 Mon 01:35:47:508.445 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-qos-config -> self 2025-09-01 Mon 01:35:47:508.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-config : self -> virtual-machine-interface-qos-config 2025-09-01 Mon 01:35:47:508.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-config : virtual-machine-interface-qos-config -> nil 2025-09-01 Mon 01:35:47:508.462 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-logging-object -> self 2025-09-01 Mon 01:35:47:508.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : self -> virtual-machine-interface-security-logging-object 2025-09-01 Mon 01:35:47:508.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : virtual-machine-interface-security-logging-object -> nil 2025-09-01 Mon 01:35:47:508.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-09-01 Mon 01:35:47:508.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.497 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:35:47:508.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-09-01 Mon 01:35:47:508.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-09-01 Mon 01:35:47:508.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.528 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:35:47:508.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : virtual-machine-interface-bridge-domain -> nil 2025-09-01 Mon 01:35:47:508.537 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.541 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-tag -> self 2025-09-01 Mon 01:35:47:508.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> virtual-machine-interface-tag 2025-09-01 Mon 01:35:47:508.549 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-interface-tag 2025-09-01 Mon 01:35:47:508.553 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.569 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.575 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:508.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.584 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:35:47:508.588 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-tag -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.596 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> virtual-network-tag 2025-09-01 Mon 01:35:47:508.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> virtual-network-tag 2025-09-01 Mon 01:35:47:508.604 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.612 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.615 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.619 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.623 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:508.628 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.632 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-09-01 Mon 01:35:47:508.636 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:35:47:508.640 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:35:47:508.644 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-09-01 Mon 01:35:47:508.648 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-09-01 Mon 01:35:47:508.651 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.655 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.661 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.665 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.672 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:508.678 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-09-01 Mon 01:35:47:508.686 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:35:47:508.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:35:47:508.693 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:35:47:508.697 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:35:47:508.701 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-09-01 Mon 01:35:47:508.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-09-01 Mon 01:35:47:508.708 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.720 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.727 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:508.732 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.738 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : project-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap project : self -> project-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.747 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap project : project-tag -> project-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> project-tag 2025-09-01 Mon 01:35:47:508.754 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : application-policy-set-tag -> project-tag 2025-09-01 Mon 01:35:47:508.758 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:35:47:508.766 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.769 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.773 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:508.777 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:508.781 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.785 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> self 2025-09-01 Mon 01:35:47:508.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-09-01 Mon 01:35:47:508.793 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-09-01 Mon 01:35:47:508.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-09-01 Mon 01:35:47:508.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> nil 2025-09-01 Mon 01:35:47:508.805 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:35:47:508.809 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : logical-router-interface -> self 2025-09-01 Mon 01:35:47:508.813 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router : self -> logical-router-interface 2025-09-01 Mon 01:35:47:508.817 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-router : logical-router-interface -> nil 2025-09-01 Mon 01:35:47:508.823 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for physical-interface 2025-09-01 Mon 01:35:47:508.827 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : physical-router-physical-interface -> self 2025-09-01 Mon 01:35:47:508.831 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:35:47:508.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for physical-interface 2025-09-01 Mon 01:35:47:508.843 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : virtual-port-group-physical-interface -> self 2025-09-01 Mon 01:35:47:508.848 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-port-group-physical-interface : self -> virtual-port-group-physical-interface 2025-09-01 Mon 01:35:47:508.852 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-port-group-physical-interface : virtual-port-group-physical-interface -> virtual-port-group-physical-interface 2025-09-01 Mon 01:35:47:508.855 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-port-group : self -> virtual-port-group-physical-interface 2025-09-01 Mon 01:35:47:508.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.864 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:35:47:508.868 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> self 2025-09-01 Mon 01:35:47:508.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : self -> physical-interface-logical-interface 2025-09-01 Mon 01:35:47:508.875 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-09-01 Mon 01:35:47:508.879 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:35:47:508.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:35:47:508.890 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-interface : logical-interface-virtual-machine-interface -> self 2025-09-01 Mon 01:35:47:508.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> logical-interface-virtual-machine-interface 2025-09-01 Mon 01:35:47:508.898 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> nil 2025-09-01 Mon 01:35:47:508.905 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:35:47:508.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap logical-interface : physical-router-logical-interface -> self 2025-09-01 Mon 01:35:47:508.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : self -> physical-router-logical-interface 2025-09-01 Mon 01:35:47:508.916 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-logical-interface -> nil 2025-09-01 Mon 01:35:47:508.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for physical-router 2025-09-01 Mon 01:35:47:508.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-router : physical-router-physical-interface -> self 2025-09-01 Mon 01:35:47:508.929 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : self -> physical-router-physical-interface 2025-09-01 Mon 01:35:47:508.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap physical-interface : physical-router-physical-interface -> nil 2025-09-01 Mon 01:35:47:508.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for service-health-check 2025-09-01 Mon 01:35:47:508.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-09-01 Mon 01:35:47:508.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-09-01 Mon 01:35:47:508.953 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> service-port-health-check 2025-09-01 Mon 01:35:47:508.957 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:508.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:35:47:508.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:35:47:508.969 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for service-health-check 2025-09-01 Mon 01:35:47:508.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-09-01 Mon 01:35:47:508.977 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-09-01 Mon 01:35:47:508.981 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> service-port-health-check 2025-09-01 Mon 01:35:47:508.995 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-09-01 Mon 01:35:47:509.000 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-09-01 Mon 01:35:47:509.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-09-01 Mon 01:35:47:509.011 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> port-tuple-interface 2025-09-01 Mon 01:35:47:509.015 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:35:47:509.019 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:35:47:509.023 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:35:47:509.027 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for qos-config 2025-09-01 Mon 01:35:47:509.031 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-config : global-qos-config-qos-config -> self 2025-09-01 Mon 01:35:47:509.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap global-qos-config : global-qos-config-qos-config -> nil 2025-09-01 Mon 01:35:47:509.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for forwarding-class 2025-09-01 Mon 01:35:47:509.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap forwarding-class : forwarding-class-qos-queue -> self 2025-09-01 Mon 01:35:47:509.049 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-queue : self -> forwarding-class-qos-queue 2025-09-01 Mon 01:35:47:509.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap qos-queue : forwarding-class-qos-queue -> nil 2025-09-01 Mon 01:35:47:509.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:35:47:509.063 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : security-logging-object-network-policy -> self 2025-09-01 Mon 01:35:47:509.069 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap network-policy : self -> security-logging-object-network-policy 2025-09-01 Mon 01:35:47:509.075 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap network-policy : security-logging-object-network-policy -> nil 2025-09-01 Mon 01:35:47:509.081 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:35:47:509.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : security-logging-object-security-group -> self 2025-09-01 Mon 01:35:47:509.092 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-group : self -> security-logging-object-security-group 2025-09-01 Mon 01:35:47:509.096 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-group : security-logging-object-security-group -> nil 2025-09-01 Mon 01:35:47:509.101 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:35:47:509.107 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : firewall-policy-security-logging-object -> self 2025-09-01 Mon 01:35:47:509.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-security-logging-object : self -> firewall-policy-security-logging-object 2025-09-01 Mon 01:35:47:509.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-security-logging-object : firewall-policy-security-logging-object -> firewall-policy-security-logging-object 2025-09-01 Mon 01:35:47:509.119 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : firewall-policy-security-logging-object -> nil 2025-09-01 Mon 01:35:47:509.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:35:47:509.127 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap security-logging-object : firewall-rule-security-logging-object -> self 2025-09-01 Mon 01:35:47:509.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule-security-logging-object : self -> firewall-rule-security-logging-object 2025-09-01 Mon 01:35:47:509.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule-security-logging-object : firewall-rule-security-logging-object -> firewall-rule-security-logging-object 2025-09-01 Mon 01:35:47:509.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : firewall-rule-security-logging-object -> nil 2025-09-01 Mon 01:35:47:509.145 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for virtual-router 2025-09-01 Mon 01:35:47:509.149 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-router : virtual-router-sub-cluster -> self 2025-09-01 Mon 01:35:47:509.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap sub-cluster : self -> virtual-router-sub-cluster 2025-09-01 Mon 01:35:47:509.157 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap sub-cluster : bgp-router-sub-cluster -> virtual-router-sub-cluster 2025-09-01 Mon 01:35:47:509.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-router : self -> bgp-router-sub-cluster 2025-09-01 Mon 01:35:47:509.165 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bgp-router : bgp-router-sub-cluster -> nil 2025-09-01 Mon 01:35:47:509.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for bridge-domain 2025-09-01 Mon 01:35:47:509.175 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-09-01 Mon 01:35:47:509.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:35:47:509.182 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-bridge-domain 2025-09-01 Mon 01:35:47:509.186 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:35:47:509.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:35:47:509.196 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for bridge-domain 2025-09-01 Mon 01:35:47:509.200 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-09-01 Mon 01:35:47:509.204 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:35:47:509.208 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-09-01 Mon 01:35:47:509.212 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for application-policy-set 2025-09-01 Mon 01:35:47:509.216 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> self 2025-09-01 Mon 01:35:47:509.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:509.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:509.227 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:35:47:509.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:35:47:509.236 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:35:47:509.240 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:35:47:509.245 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : firewall-rule-service-group -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.261 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-group : self -> firewall-rule-service-group 2025-09-01 Mon 01:35:47:509.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap service-group : firewall-rule-service-group -> nil 2025-09-01 Mon 01:35:47:509.269 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:35:47:509.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:35:47:509.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.283 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.291 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : firewall-rule-address-group -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap address-group : self -> firewall-rule-address-group 2025-09-01 Mon 01:35:47:509.299 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap address-group : address-group-tag -> firewall-rule-address-group 2025-09-01 Mon 01:35:47:509.303 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> address-group-tag 2025-09-01 Mon 01:35:47:509.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : address-group-tag -> nil 2025-09-01 Mon 01:35:47:509.311 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:35:47:509.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:35:47:509.319 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.326 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : firewall-rule-tag -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.334 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : self -> firewall-rule-tag 2025-09-01 Mon 01:35:47:509.338 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap tag : firewall-rule-tag -> nil 2025-09-01 Mon 01:35:47:509.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:35:47:509.346 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:35:47:509.349 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:35:47:509.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Adding ReactorMap firewall-rule : firewall-policy-firewall-rule -> nil 2025-09-01 Mon 01:35:47:619.211 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.0.0.22:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:35:47:624.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Current receive sock buffer size is 262144 2025-09-01 Mon 01:35:47:624.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640039802368, Pid 145094]: KsyncTxQueue CPU pinning policy <>. KsyncTxQueuen not pinned to CPU 2025-09-01 Mon 01:35:47:625.909 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Vrouter family is 32 2025-09-01 Mon 01:35:47:626.918 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Flow table size : 20185088 2025-09-01 Mon 01:35:47:626.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Flow table size : 161218560 2025-09-01 Mon 01:35:47:626.975 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SetHugePages: Bridge table size:20185088 Flow table size:161218560 2025-09-01 Mon 01:35:47:627.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Mem mapping hugepage file:/dev/hugepages2M/bridge size:20185088 2025-09-01 Mon 01:35:47:627.145 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Mem mapped hugepage file:/dev/hugepages2M/bridge to addr:0x7fe94d800000 2025-09-01 Mon 01:35:47:627.158 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Mem mapping hugepage file:/dev/hugepages2M/flow size:161218560 2025-09-01 Mon 01:35:47:627.164 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Mem mapped hugepage file:/dev/hugepages2M/flow to addr:0x7fe943e00000 2025-09-01 Mon 01:35:47:627.173 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Sending Huge Page configuration to VROUTER 2025-09-01 Mon 01:35:47:682.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Huge pages set successfully 2025-09-01 Mon 01:35:47:697.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Flow table size : 161218560 2025-09-01 Mon 01:35:47:697.872 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Flow table size : 20185088 2025-09-01 Mon 01:35:47:748.895 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: ROLE : Generator 2025-09-01 Mon 01:35:47:748.931 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: MODULE : contrail-vrouter-agent 2025-09-01 Mon 01:35:47:748.936 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: SOURCE : an-jenkins-deploy-platform-juju-k8s-1454-1 2025-09-01 Mon 01:35:47:748.940 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: NODE TYPE : Compute 2025-09-01 Mon 01:35:47:748.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: INSTANCE ID : 0 2025-09-01 Mon 01:35:47:748.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: HTTP SERVER PORT : 8085 2025-09-01 Mon 01:35:47:748.954 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-09-01 Mon 01:35:47:749.402 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: HTTP Introspect Init 2025-09-01 Mon 01:35:47:749.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8085 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:35:47:749.668 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Sandesh Http Server Port: 8085 2025-09-01 Mon 01:35:47:749.738 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: NOT Writing http_port 8085TO : /tmp/contrail-vrouter-agent.144520.http_port 2025-09-01 Mon 01:35:47:749.744 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: CONNECT TO COLLECTOR: 1 2025-09-01 Mon 01:35:47:750.131 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Idle 2025-09-01 Mon 01:35:47:751.730 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: No Client: 1756690547751186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:47:751.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvStart in state Idle 2025-09-01 Mon 01:35:47:751.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Disconnect 2025-09-01 Mon 01:35:47:752.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvCollectorUpdate in state Disconnect 2025-09-01 Mon 01:35:47:752.262 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:35:47:752.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:752.323 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547751919 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:35:47:752.336 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:47:752.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:752.397 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547751980 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:47:752.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:47:752.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:752.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547752042 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:47:752.479 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:47:752.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:752.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547752060 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:35:47:752.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:47:752.511 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:752.571 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547752296 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:47:752.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:47:752.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:754.010 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface lo flags 65609 2025-09-01 Mon 01:35:47:754.078 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface ens3 flags 69699 2025-09-01 Mon 01:35:47:754.101 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface fan-252 flags 69699 2025-09-01 Mon 01:35:47:754.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface ftun0 flags 69699 2025-09-01 Mon 01:35:47:754.129 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface lxdbr0 flags 4099 2025-09-01 Mon 01:35:47:754.135 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface docker0 flags 4099 2025-09-01 Mon 01:35:47:754.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface pkt1 flags 65601 2025-09-01 Mon 01:35:47:754.157 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface pkt3 flags 65601 2025-09-01 Mon 01:35:47:754.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface pkt2 flags 65601 2025-09-01 Mon 01:35:47:754.170 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface vhost0 flags 69699 2025-09-01 Mon 01:35:47:754.174 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink interface message RTM_NEWLINK for interface pkt0 flags 69699 2025-09-01 Mon 01:35:47:754.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface lo 2025-09-01 Mon 01:35:47:754.219 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-09-01 Mon 01:35:47:754.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface lxdbr0 2025-09-01 Mon 01:35:47:754.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface docker0 2025-09-01 Mon 01:35:47:754.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-09-01 Mon 01:35:47:754.263 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface lo 2025-09-01 Mon 01:35:47:754.270 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-09-01 Mon 01:35:47:754.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface ftun0 2025-09-01 Mon 01:35:47:754.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-09-01 Mon 01:35:47:754.292 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink address message RTM_NEWADDR for interface pkt0 2025-09-01 Mon 01:35:47:754.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547752556 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:__default__ index = 0 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-09-01 Mon 01:35:47:754.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-09-01 Mon 01:35:47:754.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:754.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547752581 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:ip-fabric index = 1 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-09-01 Mon 01:35:47:754.374 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-09-01 Mon 01:35:47:754.359 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-09-01 Mon 01:35:47:754.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-09-01 Mon 01:35:47:754.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink route message RTM_NEWROUTE : 10.193.248.0/24 Interface lxdbr0 GW 0.0.0.0 2025-09-01 Mon 01:35:47:754.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink route message RTM_NEWROUTE : 172.17.0.0/16 Interface docker0 GW 0.0.0.0 2025-09-01 Mon 01:35:47:754.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Handle netlink route message RTM_NEWROUTE : 252.0.0.0/8 Interface fan-252 GW 0.0.0.0 2025-09-01 Mon 01:35:47:754.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.435 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.442 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.446 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.450 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.454 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.472 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.480 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.483 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:35:47:754.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:35:47:754.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.516 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:35:47:754.524 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:35:47:754.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-09-01 Mon 01:35:47:754.531 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-09-01 Mon 01:35:47:754.535 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:35:47:754.539 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:35:47:754.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:754.546 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:754.550 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:754.554 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:754.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:754.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:35:47:769.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:9091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:35:47:770.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Router ID configured. Connection to controller initiated 2025-09-01 Mon 01:35:47:774.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:774.288 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690547774187 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:774.303 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:774.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:774.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690547774240 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, Collector connection down, No Configuration for self Number of connections:2, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:35:47:774.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:47:774.629 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:774.641 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690547774619 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:774.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:774.652 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:774.665 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690547774644 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:47:774.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:47:775.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_ERR]: TcpServerMessageLog: Server TCP bind(fe80::f816:3eff:fe11:8487%vhost0:8097): Cannot assign requested address src/contrail-common/io/tcp_server.cc 110 2025-09-01 Mon 01:35:47:822.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:822.971 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547822875 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:822.980 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:823.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:823.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690547823300 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40599 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:823.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:823.727 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:47:824.584 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:47:824.698 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:824.717 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690547824625 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:824.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:825.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:825.779 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690547825747 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:825.785 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:825.860 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:825.874 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690547825848 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38545 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:825.879 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:47:826.002 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:47:826.035 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:47:826.069 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:47:826.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690547826063 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:47:826.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:752.344 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: 10.0.0.50:8086 EvConnectTimerExpired in state Connect 2025-09-01 Mon 01:35:48:752.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvConnectTimerExpired in state Connect 2025-09-01 Mon 01:35:48:752.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Connect : EvConnectTimerExpired 2025-09-01 Mon 01:35:48:753.098 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Idle 2025-09-01 Mon 01:35:48:753.187 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:753.212 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690548752437 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:48:753.221 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:48:753.232 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:35:48:753.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:753.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690548753170 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 7, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:48:753.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:48:762.214 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:762.251 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690548762129 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:762.258 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:762.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:762.983 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690548762948 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40177 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:763.000 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:763.218 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:48:763.260 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:48:763.320 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:763.331 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690548763281 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:763.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:886.892 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:886.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690548886745 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:886.977 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:887.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:887.855 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690548887823 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35513 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:887.863 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:48:888.084 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:48:888.126 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:48:888.218 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:48:888.235 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690548888158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:48:888.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:49:753.191 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:35:49:754.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:35:49:754.271 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:49:754.302 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690549753275 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:49:754.325 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:49:754.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:49:754.428 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:35:49:754.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:35:49:754.466 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690549754254 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 11, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:49:754.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:49:754.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:35:49:754.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:35:49:754.523 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:49:754.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Idle 2025-09-01 Mon 01:35:49:754.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:49:755.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690549754508 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:49:755.585 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:49:755.596 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:35:49:755.627 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:49:755.702 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690549754593 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 12, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:49:755.714 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:50:617.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:617.483 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690550617389 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:617.504 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:50:617.947 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:617.984 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690550617920 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35623 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:618.019 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:50:618.118 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:50:618.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:50:618.214 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:618.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690550618204 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:618.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:50:742.329 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:742.386 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690550742293 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:742.397 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:50:742.816 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:742.851 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690550742793 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45707 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:742.858 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:50:742.970 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:50:743.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:50:743.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:50:743.103 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690550743074 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:50:743.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:51:179.895 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:44030 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:35:51:754.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:35:51:755.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:35:51:755.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:51:755.435 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690551755001 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:51:755.445 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:51:755.454 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:51:755.456 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:35:51:755.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:35:51:755.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690551755335 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 17, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:51:755.584 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:51:755.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:35:51:755.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:35:51:755.638 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:51:755.665 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Idle 2025-09-01 Mon 01:35:51:755.710 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:51:755.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690551755619 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:51:755.728 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:51:755.734 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:35:51:755.760 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:51:755.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690551755698 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:51:755.848 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:54:578.420 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:578.489 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554578357 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:578.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:578.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:578.969 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690554578916 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43225 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:578.977 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:579.180 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:54:579.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:54:579.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:579.299 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554579276 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:579.375 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:579.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:579.390 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554579284 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:579.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:579.410 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:579.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554579347 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:54:579.435 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:54:595.188 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:595.230 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554595168 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:595.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:595.671 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:595.688 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690554595646 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37357 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:595.693 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:595.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:35:54:595.884 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:54:595.918 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:596.021 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554595909 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:596.029 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:596.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:596.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554595914 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:35:54:596.058 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:35:54:596.063 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:54:596.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690554595973 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:54:596.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:55:755.720 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:35:55:756.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:35:55:756.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:55:756.413 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690555755796 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:55:756.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:55:756.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:35:55:756.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690555756370 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 22, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:55:756.548 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:35:55:756.531 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:35:55:756.586 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:35:55:756.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:35:55:756.630 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:35:55:756.668 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:35:55:756.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Idle 2025-09-01 Mon 01:35:55:756.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:55:756.768 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690555756649 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:35:55:756.774 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:35:55:756.784 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:35:55:756.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:35:55:756.873 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690555756743 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:35:55:756.885 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:01:779.518 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:01:779.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690561779438 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:01:779.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:01:779.768 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:01:779.794 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690561779760 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42467 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:01:779.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:01:780.077 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:01:780.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:01:780.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:01:780.232 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690561780178 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:01:780.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:01:780.245 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:01:780.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690561780197 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:01:780.257 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:02:732.130 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:02:732.194 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690562732094 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:02:732.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:02:732.597 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:02:732.633 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690562732573 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37055 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:02:732.641 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:02:732.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:02:732.885 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:02:732.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:02:732.979 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690562732916 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:02:733.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:02:733.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:02:733.032 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690562732922 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:02:733.046 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:03:756.747 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:36:03:757.264 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:36:03:757.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:03:757.382 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690563756826 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:03:757.389 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:03:757.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:03:757.524 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690563757343 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 27, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:03:757.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:03:757.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:36:03:757.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:36:03:757.639 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:36:03:757.647 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:36:03:757.884 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:03:757.915 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Idle 2025-09-01 Mon 01:36:03:757.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:03:757.984 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690563757681 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:03:757.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:03:758.003 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:36:03:758.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:03:758.152 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690563757952 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 28, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:03:758.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:16:324.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:16:324.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690576324323 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:16:324.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:16:324.850 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:16:324.888 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690576324825 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43477 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:16:324.897 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:16:325.083 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:16:325.183 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:16:325.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:16:325.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690576325216 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:16:325.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:16:325.250 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:16:325.267 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690576325222 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:16:325.272 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:17:494.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:495.647 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690577494803 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = __UNKNOWN__ vrf_stats_list= [ [ [ name = __untitled__ diag_packet_count = 0 unknown_unicast_floods = 0 arp_packet_counts= [ from_vm_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] from_physical_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] ] offload_packet_counts= [ gro = 0 ] nh_packet_counts= [ discards = 0 resolves = 0 l3_receives = 0 l2_receives = 0 local_vm_l3_forwards = 0 local_vm_l2_forwards = 0 vrf_translates = 0 ecmp_forwards = 0 tunnel_nh_stats= [ udp_encaps = 0 mpls_over_udp_encaps = 0 mpls_over_gre_encaps = 0 vxlan_encaps = 0 ] comp_nh_stats= [ source_replication_forwards = 0 edge_replication_forwards = 0 local_vm_l3_forwards = 0 total_multicast_forwards = 0 ] ] ], ] ] ] 2025-09-01 Mon 01:36:17:495.671 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-09-01 Mon 01:36:17:496.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:497.277 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690577496470 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 interface_list= [ [ ] ] error_intf_list= [ [ ] ] no_config_intf_list= [ [ (*_iter151) = 00000000-0000-0000-0000-000000000000, ] ] vmi_count= [ active = 1 ] down_interface_count = 1 ] 2025-09-01 Mon 01:36:17:497.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 01:36:17:751.954 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:752.031 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690577751812 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 71 dequeues = 71 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690563757914 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690577496493 ] max_sm_queue_count = 5 ] 2025-09-01 Mon 01:36:17:752.052 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:17:752.061 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:752.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690577751921 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:17:752.164 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:17:754.834 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:754.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690577754805 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 xmpp_peer_list= [ [ [ ip = 10.0.0.50 status = 0 setup_time = 0 primary = 1 ], ] ] self_ip_list= [ [ (*_iter146) = 10.0.0.22, ] ] build_info = {"build-info":[{"build-version":"v0+nightly","build-time":"2025-09-01 00:13:09.128962","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10717-1.novalocal","build-id":"unknown","build-number":"unknown"}]} phy_if= [ [ [ name = ens3 mac_address = fa:16:3e:11:84:87 ], ] ] vhost_if= [ name = vhost0 mac_address = fa:16:3e:11:84:87 ] log_file = /var/log/contrail/vrouter-agent/contrail-vrouter-agent.log config_file = /etc/contrail/contrail-vrouter-agent.conf log_local = 1 log_level = SYS_DEBUG log_category = sandesh_http_port = 8085 dns_servers= [ [ (*_iter149) = 10.0.0.50, ] ] vhost_cfg= [ name = vhost0 ip = 10.0.0.22 ip_prefix_len = 24 gateway = 10.0.0.1 ] eth_name= [ [ (*_iter150) = ens3, ] ] tunnel_type = MPLSoGRE hypervisor = kvm control_ip = 10.0.0.22 dns_server_list_cfg= [ [ (*_iter152) = 10.0.0.50:53, ] ] control_node_list_cfg= [ [ (*_iter153) = 10.0.0.50:5269, ] ] hostname_cfg = an-jenkins-deploy-platform-juju-k8s-1454-1 flow_cache_timeout_cfg = 0 ll_max_system_flows_cfg = 16384 ll_max_vm_flows_cfg = 16384 gateway_cfg_list= [ [ ] ] headless_mode_cfg = 1 collector_server_list_cfg= [ [ (*_iter155) = 10.0.0.50:8086, ] ] log_flow = 0 mode = VROUTER platform = HOST vr_limits= [ max_labels = 5120 max_nexthops = 524288 max_interfaces = 4355 max_vrfs = 4096 max_mirror_entries = 255 vrouter_max_bridge_entries = 262144 vrouter_max_oflow_bridge_entries = 53248 vrouter_build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-09-01 00:11:15.602301", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-10717-1.novalocal"}]} vrouter_max_flow_entries = 524288 vrouter_max_oflow_entries = 105472 vrouter_priority_tagging = 1 ] bgpaas_enabled = 0 port_mirror_enabled = 0 subcluster_name = vr_high_watermark = 80 vr_low_watermark = 75 vr_limit_exceeded_list= [ [ ] ] res_limit = 0 res_table_limit = 0 loopback_ip = 0.0.0.0 gateway_list= [ [ (*_iter161) = 10.0.0.1, ] ] ] 2025-09-01 Mon 01:36:17:754.937 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 01:36:17:754.948 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:755.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690577754863 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 248 in_bytes = 30805 out_tpkts = 223 out_bytes = 74818 exception_packets = 16 exception_packets_dropped = 11 exception_packets_allowed = 5 total_flows = 0 aged_flows = 0 uptime = 1756690547493900 phy_if_info= [ [ _iter254->first = ens3 [ speed = -1 duplexity = 255 ], ] ] raw_vhost_stats= [ name = vhost0 in_pkts = 248 in_bytes = 30805 out_pkts = 223 out_bytes = 74818 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 208 in_bytes = 74188 out_pkts = 250 out_bytes = 30889 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:36:17:755.067 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:36:17:755.074 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:17:755.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690577754898 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 6 inet4_multicast = 0 evpn = 0 bridge = 4 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:36:17:755.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:36:19:293.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:293.151 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690579293046 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:19:293.176 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:19:293.573 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:293.622 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690579293546 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45313 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:19:293.629 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:19:293.729 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:19:293.771 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:19:293.809 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:293.821 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690579293799 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:19:293.827 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:19:293.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:293.840 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690579293805 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:19:293.845 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:19:757.979 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:36:19:758.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:36:19:758.620 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:19:758.647 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690579758072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:19:758.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:19:758.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:19:758.734 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:36:19:758.765 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:36:19:758.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690579758592 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 32, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:19:758.834 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:19:758.842 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:36:19:758.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:36:19:758.884 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:19:758.915 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Idle 2025-09-01 Mon 01:36:19:758.958 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:758.974 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690579758865 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:19:758.980 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:19:758.994 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:36:19:759.019 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:19:759.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690579758950 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 33, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:19:759.118 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:44:945.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:44:945.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690604945381 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:44:945.514 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:44:945.656 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:44:945.694 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690604945639 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42031 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:44:945.703 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:44:945.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:44:945.836 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:44:945.877 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:44:945.936 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690604945867 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:44:945.942 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:44:945.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:44:945.956 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690604945871 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:44:945.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:47:752.210 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:752.292 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690607752035 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 88 dequeues = 88 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 107 dequeues = 107 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690579758914 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690604945948 ] max_sm_queue_count = 5 ] 2025-09-01 Mon 01:36:47:752.320 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:47:752.327 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:752.436 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: SANDESH: Send FAILED: 1756690607752172 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 35, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:47:752.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640052397568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:47:755.665 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:755.748 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690607755608 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1417662136 node_type = ] meminfo= [ virt = 32745 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.155 five_min_avg = 0.21 fifteen_min_avg = 0.145 ] cpu_share = 1.29719 ] ] 2025-09-01 Mon 01:36:47:755.761 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:36:47:755.769 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:755.783 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690607755647 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32745 cpu_share = 1.29719 used_sys_mem = 2877305160 one_min_cpuload = 0.155 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:36:47:755.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:36:47:755.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:755.896 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: SANDESH: Send FAILED: 1756690607755683 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 403 in_bytes = 47779 out_tpkts = 363 out_bytes = 99089 exception_packets = 17 exception_packets_dropped = 12 raw_vhost_stats= [ name = vhost0 in_pkts = 403 in_bytes = 47779 out_pkts = 363 out_bytes = 99089 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 333 in_bytes = 97829 out_pkts = 405 out_bytes = 47863 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:36:47:755.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640056595968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:36:47:756.146 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:47:756.175 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690607756134 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 6 inet4_multicast = 0 evpn = 0 bridge = 4 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:36:47:756.188 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:36:48:994.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:48:994.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: SANDESH: Send FAILED: 1756690608993983 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:48:994.094 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:48:994.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:48:994.547 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: SANDESH: Send FAILED: 1756690608994475 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38045 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:48:994.555 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640048199168, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:48:994.729 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:36:48:994.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:48:994.809 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:48:994.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690608994801 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:48:994.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:48:994.893 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:48:994.900 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: SANDESH: Send FAILED: 1756690608994806 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:36:48:994.905 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640060794368, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:36:49:758.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:36:49:759.319 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:36:49:759.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:49:759.493 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690609759112 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:49:759.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:49:759.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:36:49:759.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:36:49:759.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:36:49:759.685 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690609759411 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 37, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 98, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:49:759.714 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:49:759.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:36:49:759.734 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:36:49:759.774 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:36:49:759.805 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Idle 2025-09-01 Mon 01:36:49:759.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:49:759.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690609759755 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:36:49:759.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:36:49:759.900 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:36:49:759.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:36:49:760.026 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690609759844 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 145094 http_port = 8085 start_time = 1756690547750144 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 100, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:36:49:760.042 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:36:51:197.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:37232 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:36:51:201.667 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:37246 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:37:12:216.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:12:216.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: SANDESH: Send FAILED: 1756690632215999 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:12:216.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640031405568, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:12:216.180 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:12:216.194 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690632216172 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35585 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:12:216.200 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:12:216.327 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640067783296, Pid 145094]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:12:216.378 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640035603968, Pid 145094]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:12:216.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:12:216.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690632216413 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:12:216.480 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:12:216.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:12:216.493 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: SANDESH: Send FAILED: 1756690632216417 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:12:216.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140640044000768, Pid 145094]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:24:892.204 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: vhost interface name : vhost0 2025-09-01 Mon 01:37:24:892.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: vhost IP Address : 10.0.0.22/24 2025-09-01 Mon 01:37:24:892.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: vhost gateway : 0.0.0.0 2025-09-01 Mon 01:37:24:892.284 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Ethernet ports : ens3 2025-09-01 Mon 01:37:24:892.288 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Loopback IP : 0.0.0.0 2025-09-01 Mon 01:37:24:892.293 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Xmpp Servers : 10.0.0.50:5269 2025-09-01 Mon 01:37:24:892.298 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Xmpp Authentication : 0 2025-09-01 Mon 01:37:24:892.303 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Cluster-Name : 2025-09-01 Mon 01:37:24:892.308 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: DNS Servers : 10.0.0.50:53 2025-09-01 Mon 01:37:24:892.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: DNS client port : 8098 2025-09-01 Mon 01:37:24:892.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: DNS timeout : 3000 2025-09-01 Mon 01:37:24:892.321 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: DNS max retries : 2 2025-09-01 Mon 01:37:24:892.325 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Xmpp Dns Authentication : 0 2025-09-01 Mon 01:37:24:892.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TSN Servers : [] 2025-09-01 Mon 01:37:24:892.334 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: COLLECTOR Servers : 10.0.0.50:8086 2025-09-01 Mon 01:37:24:892.345 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Tunnel-Type : MPLSoGRE 2025-09-01 Mon 01:37:24:892.349 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Metadata-Proxy Shared Secret: contrail 2025-09-01 Mon 01:37:24:892.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Metadata-Proxy Port : 8097 2025-09-01 Mon 01:37:24:892.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Metadata-Proxy SSL Flag : 0 2025-09-01 Mon 01:37:24:892.360 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Linklocal Max System Flows : 16384 2025-09-01 Mon 01:37:24:892.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Linklocal Max Vm Flows : 16384 2025-09-01 Mon 01:37:24:892.368 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow cache timeout : 0 2025-09-01 Mon 01:37:24:892.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Stale Interface cleanup timeout : 60 2025-09-01 Mon 01:37:24:892.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow thread count : 1 2025-09-01 Mon 01:37:24:892.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow latency limit : 0 2025-09-01 Mon 01:37:24:892.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow index-mgr sm log count : 0 2025-09-01 Mon 01:37:24:892.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow add-tokens : 50 2025-09-01 Mon 01:37:24:892.391 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow ksync-tokens : 25 2025-09-01 Mon 01:37:24:892.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow del-tokens : 16 2025-09-01 Mon 01:37:24:892.399 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow update-tokens : 16 2025-09-01 Mon 01:37:24:892.402 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Pin flow netlink task to CPU: 2025-09-01 Mon 01:37:24:892.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Maximum sessions : 100 2025-09-01 Mon 01:37:24:892.410 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Maximum session aggregates : 8 2025-09-01 Mon 01:37:24:892.414 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Maximum session endpoints : 5 2025-09-01 Mon 01:37:24:892.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Fabric SNAT hash table size : 4096 2025-09-01 Mon 01:37:24:892.422 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Flow excluding Router ID in hash :0 2025-09-01 Mon 01:37:24:892.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Agent Mode : Vrouter 2025-09-01 Mon 01:37:24:892.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Gateway Mode : None 2025-09-01 Mon 01:37:24:892.435 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: DHCP Relay Mode : 0 2025-09-01 Mon 01:37:24:892.439 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance netns cmd : /usr/local/bin/opencontrail-vrouter-netns 2025-09-01 Mon 01:37:24:892.443 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance docker cmd : /usr/local/bin/opencontrail-vrouter-docker 2025-09-01 Mon 01:37:24:892.450 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance workers : 0 2025-09-01 Mon 01:37:24:892.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance timeout : 0 2025-09-01 Mon 01:37:24:892.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance lb ssl : 2025-09-01 Mon 01:37:24:892.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Service instance lbaas auth : 2025-09-01 Mon 01:37:24:892.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Bgp as a service port range : 50000-50512 2025-09-01 Mon 01:37:24:892.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Services queue limit : 1024 2025-09-01 Mon 01:37:24:892.472 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: BGPAAS max shared sessions for service port : 4 2025-09-01 Mon 01:37:24:892.476 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Sandesh Key file : /etc/contrail/ssl/private/server-privkey.pem 2025-09-01 Mon 01:37:24:892.480 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Sandesh Cert file : /etc/contrail/ssl/certs/server.pem 2025-09-01 Mon 01:37:24:892.484 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Sandesh CA Cert : /etc/contrail/ssl/certs/ca-cert.pem 2025-09-01 Mon 01:37:24:892.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Sandesh SSL Enable : 0 2025-09-01 Mon 01:37:24:892.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Introspect SSL Enable : 0 2025-09-01 Mon 01:37:24:892.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Hypervisor mode : kvm 2025-09-01 Mon 01:37:24:892.553 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Ethernet Port Encap Type : 2025-09-01 Mon 01:37:24:892.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: Platform mode : Vrouter on host linux kernel 2025-09-01 Mon 01:37:24:892.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: VRouter [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"v0+nightly","build-time":"2025-09-01 00:13:09.128962","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10717-1.novalocal","build-id":"unknown","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-09-01 Mon 01:37:24:892.757 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Logging: DISABLED -> ENABLED 2025-09-01 Mon 01:37:24:892.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-09-01 Mon 01:37:24:914.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> self 2025-09-01 Mon 01:37:24:914.399 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:37:24:914.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:37:24:914.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-access-control-list -> self 2025-09-01 Mon 01:37:24:914.421 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap access-control-list : self -> virtual-network-access-control-list 2025-09-01 Mon 01:37:24:914.426 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap access-control-list : virtual-network-access-control-list -> nil 2025-09-01 Mon 01:37:24:914.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> self 2025-09-01 Mon 01:37:24:914.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:37:24:914.443 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> virtual-network-network-ipam 2025-09-01 Mon 01:37:24:914.447 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:37:24:914.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.455 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-qos-config -> self 2025-09-01 Mon 01:37:24:914.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-config : self -> virtual-network-qos-config 2025-09-01 Mon 01:37:24:914.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-config : virtual-network-qos-config -> nil 2025-09-01 Mon 01:37:24:914.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-security-logging-object -> self 2025-09-01 Mon 01:37:24:914.477 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : self -> virtual-network-security-logging-object 2025-09-01 Mon 01:37:24:914.481 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : virtual-network-security-logging-object -> nil 2025-09-01 Mon 01:37:24:914.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : logical-router-virtual-network -> self 2025-09-01 Mon 01:37:24:914.494 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router-virtual-network : self -> logical-router-virtual-network 2025-09-01 Mon 01:37:24:914.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router-virtual-network : logical-router-virtual-network -> logical-router-virtual-network 2025-09-01 Mon 01:37:24:914.503 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router : self -> logical-router-virtual-network 2025-09-01 Mon 01:37:24:914.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router : logical-router-virtual-network -> nil 2025-09-01 Mon 01:37:24:914.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-network 2025-09-01 Mon 01:37:24:914.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-multicast-policy -> self 2025-09-01 Mon 01:37:24:914.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap multicast-policy : self -> virtual-network-multicast-policy 2025-09-01 Mon 01:37:24:914.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap multicast-policy : virtual-network-multicast-policy -> nil 2025-09-01 Mon 01:37:24:914.530 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for routing-instance 2025-09-01 Mon 01:37:24:914.540 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> self 2025-09-01 Mon 01:37:24:914.544 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-routing-instance 2025-09-01 Mon 01:37:24:914.548 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-routing-instance 2025-09-01 Mon 01:37:24:914.552 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:37:24:914.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:37:24:914.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for security-group 2025-09-01 Mon 01:37:24:914.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-group : security-group-access-control-list -> self 2025-09-01 Mon 01:37:24:914.569 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap access-control-list : self -> security-group-access-control-list 2025-09-01 Mon 01:37:24:914.573 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap access-control-list : security-group-access-control-list -> nil 2025-09-01 Mon 01:37:24:914.578 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for tag 2025-09-01 Mon 01:37:24:914.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> self 2025-09-01 Mon 01:37:24:914.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:914.591 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : policy-management-application-policy-set -> application-policy-set-tag 2025-09-01 Mon 01:37:24:914.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap policy-management : policy-management-application-policy-set -> nil 2025-09-01 Mon 01:37:24:914.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.606 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:37:24:914.610 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:914.614 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:914.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:37:24:914.622 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:37:24:914.627 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.630 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-09-01 Mon 01:37:24:914.634 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:37:24:914.638 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : virtual-machine-interface-virtual-machine -> nil 2025-09-01 Mon 01:37:24:914.642 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-09-01 Mon 01:37:24:914.650 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:37:24:914.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> nil 2025-09-01 Mon 01:37:24:914.658 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.662 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:37:24:914.666 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:914.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:914.674 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:37:24:914.678 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:37:24:914.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.688 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-routing-instance -> self 2025-09-01 Mon 01:37:24:914.693 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-routing-instance : self -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:37:24:914.697 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-routing-instance : virtual-machine-interface-routing-instance -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:37:24:914.701 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : self -> virtual-machine-interface-routing-instance 2025-09-01 Mon 01:37:24:914.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : virtual-machine-interface-routing-instance -> nil 2025-09-01 Mon 01:37:24:914.795 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-group -> self 2025-09-01 Mon 01:37:24:914.804 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-group : self -> virtual-machine-interface-security-group 2025-09-01 Mon 01:37:24:914.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-group : virtual-machine-interface-security-group -> nil 2025-09-01 Mon 01:37:24:914.814 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.817 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : alias-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:914.822 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap alias-ip : self -> alias-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap alias-ip : alias-ip-pool-alias-ip -> alias-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap alias-ip-pool : virtual-network-alias-ip-pool -> alias-ip-pool-alias-ip 2025-09-01 Mon 01:37:24:914.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-alias-ip-pool 2025-09-01 Mon 01:37:24:914.869 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-alias-ip-pool -> nil 2025-09-01 Mon 01:37:24:914.874 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.878 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:914.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.895 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-09-01 Mon 01:37:24:914.899 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:37:24:914.903 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:37:24:914.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-provider-network 2025-09-01 Mon 01:37:24:914.911 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-provider-network -> nil 2025-09-01 Mon 01:37:24:914.916 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.919 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:914.923 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : floating-ip-pool-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.931 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip-pool : virtual-network-floating-ip-pool -> floating-ip-pool-floating-ip 2025-09-01 Mon 01:37:24:914.935 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:37:24:914.939 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-floating-ip-pool 2025-09-01 Mon 01:37:24:914.943 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:37:24:914.947 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:37:24:914.951 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.955 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : floating-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:914.959 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : self -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap floating-ip : instance-ip-floating-ip -> floating-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:914.967 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap instance-ip : instance-ip-virtual-network -> instance-ip-floating-ip 2025-09-01 Mon 01:37:24:914.971 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> instance-ip-virtual-network 2025-09-01 Mon 01:37:24:914.977 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : instance-ip-virtual-network -> nil 2025-09-01 Mon 01:37:24:914.981 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:914.993 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : instance-ip-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:914.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap instance-ip : self -> instance-ip-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap instance-ip : instance-ip-virtual-machine-interface -> nil 2025-09-01 Mon 01:37:24:915.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.009 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-route-table -> self 2025-09-01 Mon 01:37:24:915.013 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap interface-route-table : self -> virtual-machine-interface-route-table 2025-09-01 Mon 01:37:24:915.017 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap interface-route-table : virtual-machine-interface-route-table -> nil 2025-09-01 Mon 01:37:24:915.022 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.026 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : subnet-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.030 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap subnet : self -> subnet-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.034 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap subnet : subnet-virtual-machine-interface -> nil 2025-09-01 Mon 01:37:24:915.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.043 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> logical-interface-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.052 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-09-01 Mon 01:37:24:915.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:37:24:915.060 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.065 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.069 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-port-group-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.076 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> virtual-port-group-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : physical-router-physical-interface -> virtual-port-group-physical-interface 2025-09-01 Mon 01:37:24:915.083 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:37:24:915.087 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.092 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.096 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-as-a-service : bgpaas-bgp-router -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-router : self -> bgpaas-bgp-router 2025-09-01 Mon 01:37:24:915.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-router : instance-bgp-router -> bgpaas-bgp-router 2025-09-01 Mon 01:37:24:915.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : self -> instance-bgp-router 2025-09-01 Mon 01:37:24:915.120 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : instance-bgp-router -> nil 2025-09-01 Mon 01:37:24:915.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.128 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : bgpaas-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-as-a-service : self -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-as-a-service : bgpaas-health-check -> bgpaas-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-health-check : self -> bgpaas-health-check 2025-09-01 Mon 01:37:24:915.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-health-check : bgpaas-health-check -> nil 2025-09-01 Mon 01:37:24:915.149 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.152 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-qos-config -> self 2025-09-01 Mon 01:37:24:915.156 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-config : self -> virtual-machine-interface-qos-config 2025-09-01 Mon 01:37:24:915.163 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-config : virtual-machine-interface-qos-config -> nil 2025-09-01 Mon 01:37:24:915.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-security-logging-object -> self 2025-09-01 Mon 01:37:24:915.175 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : self -> virtual-machine-interface-security-logging-object 2025-09-01 Mon 01:37:24:915.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : virtual-machine-interface-security-logging-object -> nil 2025-09-01 Mon 01:37:24:915.184 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.188 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-09-01 Mon 01:37:24:915.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.196 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.201 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.205 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:37:24:915.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-09-01 Mon 01:37:24:915.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.221 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-bridge-domain -> self 2025-09-01 Mon 01:37:24:915.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface-bridge-domain : virtual-machine-interface-bridge-domain -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : self -> virtual-machine-interface-bridge-domain 2025-09-01 Mon 01:37:24:915.291 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : virtual-machine-interface-bridge-domain -> nil 2025-09-01 Mon 01:37:24:915.299 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.302 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-tag -> self 2025-09-01 Mon 01:37:24:915.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> virtual-machine-interface-tag 2025-09-01 Mon 01:37:24:915.311 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-interface-tag 2025-09-01 Mon 01:37:24:915.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.319 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.323 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.327 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:915.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.346 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> self 2025-09-01 Mon 01:37:24:915.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:915.354 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-tag -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:915.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> virtual-network-tag 2025-09-01 Mon 01:37:24:915.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> virtual-network-tag 2025-09-01 Mon 01:37:24:915.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.373 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.377 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:915.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> self 2025-09-01 Mon 01:37:24:915.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:37:24:915.404 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:37:24:915.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-09-01 Mon 01:37:24:915.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-09-01 Mon 01:37:24:915.416 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.423 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:915.439 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.443 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-sub-interface -> self 2025-09-01 Mon 01:37:24:915.447 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:37:24:915.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-machine -> virtual-machine-interface-sub-interface 2025-09-01 Mon 01:37:24:915.455 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : self -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:37:24:915.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine : virtual-machine-tag -> virtual-machine-interface-virtual-machine 2025-09-01 Mon 01:37:24:915.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> virtual-machine-tag 2025-09-01 Mon 01:37:24:915.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> virtual-machine-tag 2025-09-01 Mon 01:37:24:915.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.477 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.480 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.484 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.488 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.492 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:915.497 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : project-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap project : self -> project-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap project : project-tag -> project-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> project-tag 2025-09-01 Mon 01:37:24:915.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : application-policy-set-tag -> project-tag 2025-09-01 Mon 01:37:24:915.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : self -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> application-policy-set-tag 2025-09-01 Mon 01:37:24:915.529 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.533 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.537 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:915.542 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:915.547 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> self 2025-09-01 Mon 01:37:24:915.555 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.563 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.567 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> nil 2025-09-01 Mon 01:37:24:915.571 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-machine-interface 2025-09-01 Mon 01:37:24:915.575 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : logical-router-interface -> self 2025-09-01 Mon 01:37:24:915.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router : self -> logical-router-interface 2025-09-01 Mon 01:37:24:915.583 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-router : logical-router-interface -> nil 2025-09-01 Mon 01:37:24:915.588 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for physical-interface 2025-09-01 Mon 01:37:24:915.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : physical-router-physical-interface -> self 2025-09-01 Mon 01:37:24:915.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:37:24:915.599 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.604 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for physical-interface 2025-09-01 Mon 01:37:24:915.607 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : virtual-port-group-physical-interface -> self 2025-09-01 Mon 01:37:24:915.613 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-port-group-physical-interface : self -> virtual-port-group-physical-interface 2025-09-01 Mon 01:37:24:915.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-port-group-physical-interface : virtual-port-group-physical-interface -> virtual-port-group-physical-interface 2025-09-01 Mon 01:37:24:915.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-port-group : self -> virtual-port-group-physical-interface 2025-09-01 Mon 01:37:24:915.624 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-port-group : virtual-port-group-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:37:24:915.635 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-interface : physical-interface-logical-interface -> self 2025-09-01 Mon 01:37:24:915.639 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : self -> physical-interface-logical-interface 2025-09-01 Mon 01:37:24:915.643 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : physical-router-physical-interface -> physical-interface-logical-interface 2025-09-01 Mon 01:37:24:915.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : self -> physical-router-physical-interface 2025-09-01 Mon 01:37:24:915.650 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:37:24:915.658 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-interface : logical-interface-virtual-machine-interface -> self 2025-09-01 Mon 01:37:24:915.662 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> logical-interface-virtual-machine-interface 2025-09-01 Mon 01:37:24:915.666 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : logical-interface-virtual-machine-interface -> nil 2025-09-01 Mon 01:37:24:915.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for logical-interface 2025-09-01 Mon 01:37:24:915.674 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap logical-interface : physical-router-logical-interface -> self 2025-09-01 Mon 01:37:24:915.678 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : self -> physical-router-logical-interface 2025-09-01 Mon 01:37:24:915.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-logical-interface -> nil 2025-09-01 Mon 01:37:24:915.686 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for physical-router 2025-09-01 Mon 01:37:24:915.690 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-router : physical-router-physical-interface -> self 2025-09-01 Mon 01:37:24:915.694 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : self -> physical-router-physical-interface 2025-09-01 Mon 01:37:24:915.698 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap physical-interface : physical-router-physical-interface -> nil 2025-09-01 Mon 01:37:24:915.703 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for service-health-check 2025-09-01 Mon 01:37:24:915.706 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-09-01 Mon 01:37:24:915.710 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-09-01 Mon 01:37:24:915.718 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> service-port-health-check 2025-09-01 Mon 01:37:24:915.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:915.729 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:37:24:915.734 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:37:24:915.740 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for service-health-check 2025-09-01 Mon 01:37:24:915.745 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-health-check : service-port-health-check -> self 2025-09-01 Mon 01:37:24:915.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> service-port-health-check 2025-09-01 Mon 01:37:24:915.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : port-tuple-interface -> service-port-health-check 2025-09-01 Mon 01:37:24:915.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap port-tuple : self -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap port-tuple : port-tuple-interface -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.772 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : self -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.778 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-machine-interface : virtual-machine-interface-virtual-network -> port-tuple-interface 2025-09-01 Mon 01:37:24:915.784 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-network-ipam -> virtual-machine-interface-virtual-network 2025-09-01 Mon 01:37:24:915.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : self -> virtual-network-network-ipam 2025-09-01 Mon 01:37:24:915.792 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network-network-ipam : virtual-network-network-ipam -> nil 2025-09-01 Mon 01:37:24:915.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for qos-config 2025-09-01 Mon 01:37:24:915.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-config : global-qos-config-qos-config -> self 2025-09-01 Mon 01:37:24:915.805 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap global-qos-config : global-qos-config-qos-config -> nil 2025-09-01 Mon 01:37:24:915.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for forwarding-class 2025-09-01 Mon 01:37:24:915.814 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap forwarding-class : forwarding-class-qos-queue -> self 2025-09-01 Mon 01:37:24:915.819 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-queue : self -> forwarding-class-qos-queue 2025-09-01 Mon 01:37:24:915.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap qos-queue : forwarding-class-qos-queue -> nil 2025-09-01 Mon 01:37:24:915.830 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:37:24:915.834 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : security-logging-object-network-policy -> self 2025-09-01 Mon 01:37:24:915.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap network-policy : self -> security-logging-object-network-policy 2025-09-01 Mon 01:37:24:915.842 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap network-policy : security-logging-object-network-policy -> nil 2025-09-01 Mon 01:37:24:915.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:37:24:915.850 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : security-logging-object-security-group -> self 2025-09-01 Mon 01:37:24:915.854 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-group : self -> security-logging-object-security-group 2025-09-01 Mon 01:37:24:918.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-group : security-logging-object-security-group -> nil 2025-09-01 Mon 01:37:24:918.227 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:37:24:918.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : firewall-policy-security-logging-object -> self 2025-09-01 Mon 01:37:24:918.238 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-security-logging-object : self -> firewall-policy-security-logging-object 2025-09-01 Mon 01:37:24:918.242 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-security-logging-object : firewall-policy-security-logging-object -> firewall-policy-security-logging-object 2025-09-01 Mon 01:37:24:918.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : firewall-policy-security-logging-object -> nil 2025-09-01 Mon 01:37:24:918.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for security-logging-object 2025-09-01 Mon 01:37:24:918.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap security-logging-object : firewall-rule-security-logging-object -> self 2025-09-01 Mon 01:37:24:918.261 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule-security-logging-object : self -> firewall-rule-security-logging-object 2025-09-01 Mon 01:37:24:918.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule-security-logging-object : firewall-rule-security-logging-object -> firewall-rule-security-logging-object 2025-09-01 Mon 01:37:24:918.269 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : firewall-rule-security-logging-object -> nil 2025-09-01 Mon 01:37:24:918.276 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for virtual-router 2025-09-01 Mon 01:37:24:918.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-router : virtual-router-sub-cluster -> self 2025-09-01 Mon 01:37:24:918.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap sub-cluster : self -> virtual-router-sub-cluster 2025-09-01 Mon 01:37:24:918.294 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap sub-cluster : bgp-router-sub-cluster -> virtual-router-sub-cluster 2025-09-01 Mon 01:37:24:918.298 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-router : self -> bgp-router-sub-cluster 2025-09-01 Mon 01:37:24:918.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bgp-router : bgp-router-sub-cluster -> nil 2025-09-01 Mon 01:37:24:918.310 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for bridge-domain 2025-09-01 Mon 01:37:24:918.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-09-01 Mon 01:37:24:918.319 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:37:24:918.323 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-routing-instance -> virtual-network-bridge-domain 2025-09-01 Mon 01:37:24:918.327 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : self -> virtual-network-routing-instance 2025-09-01 Mon 01:37:24:918.331 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap routing-instance : virtual-network-routing-instance -> nil 2025-09-01 Mon 01:37:24:918.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for bridge-domain 2025-09-01 Mon 01:37:24:918.339 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap bridge-domain : virtual-network-bridge-domain -> self 2025-09-01 Mon 01:37:24:918.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : self -> virtual-network-bridge-domain 2025-09-01 Mon 01:37:24:918.347 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap virtual-network : virtual-network-bridge-domain -> nil 2025-09-01 Mon 01:37:24:918.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for application-policy-set 2025-09-01 Mon 01:37:24:918.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set : application-policy-set-firewall-policy -> self 2025-09-01 Mon 01:37:24:918.360 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:918.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap application-policy-set-firewall-policy : application-policy-set-firewall-policy -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:918.368 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : self -> application-policy-set-firewall-policy 2025-09-01 Mon 01:37:24:918.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : application-policy-set-firewall-policy -> nil 2025-09-01 Mon 01:37:24:918.380 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:37:24:918.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:37:24:918.389 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : firewall-rule-service-group -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-group : self -> firewall-rule-service-group 2025-09-01 Mon 01:37:24:918.409 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap service-group : firewall-rule-service-group -> nil 2025-09-01 Mon 01:37:24:918.414 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:37:24:918.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:37:24:918.421 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.425 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.433 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : firewall-rule-address-group -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.437 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap address-group : self -> firewall-rule-address-group 2025-09-01 Mon 01:37:24:918.441 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap address-group : address-group-tag -> firewall-rule-address-group 2025-09-01 Mon 01:37:24:918.445 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> address-group-tag 2025-09-01 Mon 01:37:24:918.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : address-group-tag -> nil 2025-09-01 Mon 01:37:24:918.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:37:24:918.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:37:24:918.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.467 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.470 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.474 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : firewall-rule-tag -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : self -> firewall-rule-tag 2025-09-01 Mon 01:37:24:918.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap tag : firewall-rule-tag -> nil 2025-09-01 Mon 01:37:24:918.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Updating dependency tacker rules for firewall-policy 2025-09-01 Mon 01:37:24:918.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy : firewall-policy-firewall-rule -> self 2025-09-01 Mon 01:37:24:918.494 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-policy-firewall-rule : firewall-policy-firewall-rule -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : self -> firewall-policy-firewall-rule 2025-09-01 Mon 01:37:24:918.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Adding ReactorMap firewall-rule : firewall-policy-firewall-rule -> nil 2025-09-01 Mon 01:37:25:212.305 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.0.0.22:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:37:25:213.263 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Current receive sock buffer size is 262144 2025-09-01 Mon 01:37:25:213.337 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575972857344, Pid 147334]: KsyncTxQueue CPU pinning policy <>. KsyncTxQueuen not pinned to CPU 2025-09-01 Mon 01:37:25:216.563 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Vrouter family is 32 2025-09-01 Mon 01:37:25:216.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Flow table size : 20185088 2025-09-01 Mon 01:37:25:240.139 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Flow table size : 161218560 2025-09-01 Mon 01:37:25:240.173 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SetHugePages: Bridge table size:20185088 Flow table size:161218560 2025-09-01 Mon 01:37:25:240.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Mem mapping hugepage file:/dev/hugepages2M/bridge size:20185088 2025-09-01 Mon 01:37:25:240.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Mem mapped hugepage file:/dev/hugepages2M/bridge to addr:0x7fda62a00000 2025-09-01 Mon 01:37:25:240.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Mem mapping hugepage file:/dev/hugepages2M/flow size:161218560 2025-09-01 Mon 01:37:25:240.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Mem mapped hugepage file:/dev/hugepages2M/flow to addr:0x7fda59000000 2025-09-01 Mon 01:37:25:240.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Sending Huge Page configuration to VROUTER 2025-09-01 Mon 01:37:25:240.280 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Vrouter using already allocated memory 2025-09-01 Mon 01:37:25:281.704 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Flow table size : 161218560 2025-09-01 Mon 01:37:25:281.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Flow table size : 20185088 2025-09-01 Mon 01:37:25:383.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: ROLE : Generator 2025-09-01 Mon 01:37:25:383.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: MODULE : contrail-vrouter-agent 2025-09-01 Mon 01:37:25:383.483 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: SOURCE : an-jenkins-deploy-platform-juju-k8s-1454-1 2025-09-01 Mon 01:37:25:383.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: NODE TYPE : Compute 2025-09-01 Mon 01:37:25:383.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: INSTANCE ID : 0 2025-09-01 Mon 01:37:25:383.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: HTTP SERVER PORT : 8085 2025-09-01 Mon 01:37:25:383.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-09-01 Mon 01:37:25:383.957 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: HTTP Introspect Init 2025-09-01 Mon 01:37:25:384.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8085 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:37:25:384.175 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Sandesh Http Server Port: 8085 2025-09-01 Mon 01:37:25:384.218 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: NOT Writing http_port 8085TO : /tmp/contrail-vrouter-agent.146222.http_port 2025-09-01 Mon 01:37:25:384.223 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: CONNECT TO COLLECTOR: 1 2025-09-01 Mon 01:37:25:384.518 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Idle 2025-09-01 Mon 01:37:25:384.661 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: No Client: 1756690645384600 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:25:385.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvStart in state Idle 2025-09-01 Mon 01:37:25:385.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Disconnect 2025-09-01 Mon 01:37:25:385.415 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Disconnect 2025-09-01 Mon 01:37:25:385.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645385365 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:37:25:385.444 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Disconnect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:25:385.450 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Disconnect 2025-09-01 Mon 01:37:25:385.497 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645385408 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:25:385.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Disconnect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:25:390.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvCollectorUpdate in state Disconnect 2025-09-01 Mon 01:37:25:390.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:25:390.413 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:390.475 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645390158 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Disconnect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 2, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:25:390.489 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:25:390.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:390.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645390183 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Collector connection down, No Configuration for self Number of connections:1, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:37:25:390.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:25:390.519 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:390.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645390398 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 2, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:25:390.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:25:391.316 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:391.347 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645390698 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:__default__ index = 0 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-09-01 Mon 01:37:25:391.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-09-01 Mon 01:37:25:392.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:392.437 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645391390 AgentVrf [SYS_INFO]: VrfObjectLog: vrf= [ event = Addition name = default-domain:default-project:ip-fabric:ip-fabric index = 1 ] file = controller/src/vnsw/agent/oper/vrf.cc line = 414 2025-09-01 Mon 01:37:25:392.444 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: VrfObjectLog 2025-09-01 Mon 01:37:25:393.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface lo flags 65609 2025-09-01 Mon 01:37:25:393.701 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface ens3 flags 69699 2025-09-01 Mon 01:37:25:393.732 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface fan-252 flags 69699 2025-09-01 Mon 01:37:25:393.736 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface ftun0 flags 69699 2025-09-01 Mon 01:37:25:393.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface lxdbr0 flags 4099 2025-09-01 Mon 01:37:25:393.764 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface docker0 flags 4099 2025-09-01 Mon 01:37:25:393.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface pkt1 flags 65601 2025-09-01 Mon 01:37:25:393.786 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface pkt3 flags 65601 2025-09-01 Mon 01:37:25:393.791 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface pkt2 flags 65601 2025-09-01 Mon 01:37:25:393.799 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface vhost0 flags 69699 2025-09-01 Mon 01:37:25:393.803 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink interface message RTM_NEWLINK for interface pkt0 flags 69699 2025-09-01 Mon 01:37:25:393.836 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface lo 2025-09-01 Mon 01:37:25:393.844 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-09-01 Mon 01:37:25:393.851 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface lxdbr0 2025-09-01 Mon 01:37:25:393.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface docker0 2025-09-01 Mon 01:37:25:393.868 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-09-01 Mon 01:37:25:393.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface lo 2025-09-01 Mon 01:37:25:393.897 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface fan-252 2025-09-01 Mon 01:37:25:393.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface ftun0 2025-09-01 Mon 01:37:25:393.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface vhost0 2025-09-01 Mon 01:37:25:393.921 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink address message RTM_NEWADDR for interface pkt0 2025-09-01 Mon 01:37:25:393.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:393.970 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-09-01 Mon 01:37:25:393.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink route message RTM_NEWROUTE : 10.0.0.0/24 Interface vhost0 GW 0.0.0.0 2025-09-01 Mon 01:37:25:393.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink route message RTM_NEWROUTE : 10.193.248.0/24 Interface lxdbr0 GW 0.0.0.0 2025-09-01 Mon 01:37:25:394.009 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.017 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink route message RTM_NEWROUTE : 172.17.0.0/16 Interface docker0 GW 0.0.0.0 2025-09-01 Mon 01:37:25:394.024 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Handle netlink route message RTM_NEWROUTE : 252.0.0.0/8 Interface fan-252 GW 0.0.0.0 2025-09-01 Mon 01:37:25:394.029 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.037 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.044 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.052 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.056 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.060 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.064 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.067 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.071 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.075 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.079 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.083 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 2 scope 254 2025-09-01 Mon 01:37:25:394.087 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 2 table 255 type 3 scope 253 2025-09-01 Mon 01:37:25:394.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.109 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.113 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 254 type 1 scope 0 2025-09-01 Mon 01:37:25:394.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:37:25:394.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-09-01 Mon 01:37:25:394.128 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-09-01 Mon 01:37:25:394.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 4 scope 0 2025-09-01 Mon 01:37:25:394.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:37:25:394.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:37:25:394.143 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 2 scope 0 2025-09-01 Mon 01:37:25:394.147 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:394.151 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:394.155 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:394.159 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:394.163 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:394.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Ignoring Netlink route with family 10 table 255 type 5 scope 0 2025-09-01 Mon 01:37:25:443.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:9091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:37:25:443.758 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Router ID configured. Connection to controller initiated 2025-09-01 Mon 01:37:25:450.187 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:450.226 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645444291 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:450.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:453.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:453.892 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645450302 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, Collector connection down, No Configuration for self Number of connections:2, Expected: 3 ], ] ] ] 2025-09-01 Mon 01:37:25:453.901 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:25:454.936 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:454.952 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645454352 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:454.959 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:455.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:455.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645454998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:25:455.602 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:25:463.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server [fe80::f816:3eff:fe11:8487%vhost0]:8097 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-01 Mon 01:37:25:510.034 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:510.078 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645509825 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:510.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:510.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:510.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690645510015 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43637 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:510.120 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:510.796 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:25:510.865 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:25:510.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:510.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690645510905 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:510.950 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:512.148 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:512.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690645512135 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:512.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:25:512.223 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:25:512.235 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690645512219 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 46099 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:25:512.240 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:26:391.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: 10.0.0.50:8086 EvConnectTimerExpired in state Connect 2025-09-01 Mon 01:37:26:391.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvConnectTimerExpired in state Connect 2025-09-01 Mon 01:37:26:391.282 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvConnectTimerExpired 2025-09-01 Mon 01:37:26:391.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:37:26:391.452 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:391.474 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690646391325 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:26:391.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:26:391.494 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:26:391.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:391.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690646391432 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 5, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:26:391.676 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:26:412.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:412.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690646412351 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvConnectTimerExpired close_reason = Connect timer expired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:26:412.446 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:26:537.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:537.331 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690646537094 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:26:537.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:26:538.013 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:538.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690646537993 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40913 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:26:538.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:26:538.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:26:538.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:26:538.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:26:538.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690646538292 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:26:538.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:27:391.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:37:27:392.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:27:392.616 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:27:392.653 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690647391497 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:27:392.685 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:37:27:392.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:37:27:392.709 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:27:392.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:27:392.849 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690647392601 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 7, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:27:392.869 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:27:392.881 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:37:27:392.886 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:37:27:392.923 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:27:392.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Idle 2025-09-01 Mon 01:37:27:393.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:27:393.051 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690647392906 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:27:393.060 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:27:393.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:27:393.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:27:393.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690647393025 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:27:393.183 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:27:429.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:27:429.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690647429737 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:27:429.974 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:27:429.998 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:27:430.011 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690647429895 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33691 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:27:430.017 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:27:430.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:27:430.142 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:27:430.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:27:430.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690647430168 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:27:430.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:28:410.555 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:28:410.613 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690648410390 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:28:410.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:28:411.514 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:28:411.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690648411504 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33745 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:28:411.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:28:411.735 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:28:411.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:28:411.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:28:411.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690648411859 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:28:411.939 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:29:302.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:29:302.442 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690649302276 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:29:302.452 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:29:303.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:29:303.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690649303355 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45251 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:29:303.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:29:303.730 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:29:303.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:29:303.903 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:29:303.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690649303836 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:29:303.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:29:392.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:37:29:393.297 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:29:393.377 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:29:393.399 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690649393082 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:29:393.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:29:393.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:29:393.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690649393367 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 14, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:29:393.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:37:29:393.572 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:37:29:393.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:29:394.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:37:29:394.373 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:37:29:394.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:29:394.432 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Idle 2025-09-01 Mon 01:37:29:394.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:29:394.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690649394391 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:29:394.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:29:394.514 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:29:394.542 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:29:394.642 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690649394476 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 15, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:29:394.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:31:207.339 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.22:46554 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:37:32:192.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:32:192.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690652191958 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:32:192.170 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:32:193.025 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:32:193.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690652193011 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37719 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:32:193.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:32:193.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:32:193.280 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:32:193.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:32:193.374 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690652193304 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:32:193.380 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:32:193.385 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:32:193.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690652193309 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:32:193.397 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:32:193.454 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:32:193.472 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690652193438 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:32:193.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:33:228.087 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:228.139 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690653227949 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:33:228.147 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:33:229.052 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:229.074 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690653229036 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35097 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:33:229.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:33:229.292 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:33:229.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:33:229.956 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:229.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653229354 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:33:229.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:33:229.993 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:230.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653229360 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:33:230.006 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:33:230.011 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:230.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653229476 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:33:230.060 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:33:400.870 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:37:33:401.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:33:402.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:33:402.128 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653400944 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:33:402.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:33:402.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:33:402.242 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:37:33:402.250 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:37:33:402.281 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653402083 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 20, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:33:402.299 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:33:402.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:37:33:402.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:37:33:402.344 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:33:402.368 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:37:33:402.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:402.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653402329 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:33:402.423 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:33:402.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:33:402.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:33:402.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690653402396 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 21, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:33:402.542 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:39:681.516 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:39:682.385 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690659681410 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:39:682.413 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:39:682.588 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:39:682.640 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:39:682.777 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690659682610 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33415 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:39:682.784 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:39:682.917 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:39:683.022 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:39:683.034 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690659682940 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:39:683.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:39:683.046 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:39:683.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690659682944 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:39:683.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:41:402.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:37:41:402.713 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:41:402.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:41:402.826 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690661402509 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:41:402.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:41:402.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:41:402.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:37:41:402.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:37:41:403.017 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690661402786 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:41:403.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:41:403.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:37:41:403.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:37:41:403.089 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:41:403.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:37:41:403.156 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:403.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690661403074 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:41:403.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:41:403.191 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:41:403.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:403.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690661403148 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 24, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:41:403.305 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:41:581.567 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:581.644 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690661581458 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:41:581.653 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:41:582.474 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:582.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690661582461 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37041 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:41:582.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:41:582.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:41:582.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:41:582.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:582.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690661582742 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:41:582.821 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:41:582.826 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:41:582.834 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690661582746 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:41:582.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:51:222.899 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:41210 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:37:54:893.065 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:54:893.143 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690674893037 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 interface_list= [ [ ] ] error_intf_list= [ [ ] ] no_config_intf_list= [ [ (*_iter151) = 00000000-0000-0000-0000-000000000000, ] ] vmi_count= [ active = 1 ] down_interface_count = 1 ] 2025-09-01 Mon 01:37:54:893.155 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 01:37:54:893.759 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:54:893.799 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690674893692 [SYS_NOTICE]: UveVirtualNetworkAgentTrace: data= [ name = __UNKNOWN__ vrf_stats_list= [ [ [ name = __untitled__ diag_packet_count = 0 unknown_unicast_floods = 0 arp_packet_counts= [ from_vm_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] from_physical_interface= [ stats= [ proxies = 0 stitches = 0 floods = 0 ] ] ] offload_packet_counts= [ gro = 0 ] nh_packet_counts= [ discards = 0 resolves = 0 l3_receives = 0 l2_receives = 0 local_vm_l3_forwards = 0 local_vm_l2_forwards = 0 vrf_translates = 0 ecmp_forwards = 0 tunnel_nh_stats= [ udp_encaps = 0 mpls_over_udp_encaps = 0 mpls_over_gre_encaps = 0 vxlan_encaps = 0 ] comp_nh_stats= [ source_replication_forwards = 0 edge_replication_forwards = 0 local_vm_l3_forwards = 0 total_multicast_forwards = 0 ] ] ], ] ] ] 2025-09-01 Mon 01:37:54:893.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: UveVirtualNetworkAgentTrace 2025-09-01 Mon 01:37:55:384.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:384.922 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690675384716 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 67 dequeues = 67 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690661403117 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690674893756 ] max_sm_queue_count = 3 ] 2025-09-01 Mon 01:37:55:384.935 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:55:384.950 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:385.064 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690675384821 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 27, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:55:385.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:55:394.497 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:394.570 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690675394422 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 xmpp_peer_list= [ [ [ ip = 10.0.0.50 status = 0 setup_time = 0 primary = 1 ], ] ] self_ip_list= [ [ (*_iter146) = 10.0.0.22, ] ] build_info = {"build-info":[{"build-version":"v0+nightly","build-time":"2025-09-01 00:13:09.128962","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10717-1.novalocal","build-id":"unknown","build-number":"unknown"}]} phy_if= [ [ [ name = ens3 mac_address = fa:16:3e:11:84:87 ], ] ] vhost_if= [ name = vhost0 mac_address = fa:16:3e:11:84:87 ] log_file = /var/log/contrail/vrouter-agent/contrail-vrouter-agent.log config_file = /etc/contrail/contrail-vrouter-agent.conf log_local = 1 log_level = SYS_DEBUG log_category = sandesh_http_port = 8085 dns_servers= [ [ (*_iter149) = 10.0.0.50, ] ] vhost_cfg= [ name = vhost0 ip = 10.0.0.22 ip_prefix_len = 24 gateway = 10.0.0.1 ] eth_name= [ [ (*_iter150) = ens3, ] ] tunnel_type = MPLSoGRE hypervisor = kvm control_ip = 10.0.0.22 dns_server_list_cfg= [ [ (*_iter152) = 10.0.0.50:53, ] ] control_node_list_cfg= [ [ (*_iter153) = 10.0.0.50:5269, ] ] hostname_cfg = an-jenkins-deploy-platform-juju-k8s-1454-1 flow_cache_timeout_cfg = 0 ll_max_system_flows_cfg = 16384 ll_max_vm_flows_cfg = 16384 gateway_cfg_list= [ [ ] ] headless_mode_cfg = 1 collector_server_list_cfg= [ [ (*_iter155) = 10.0.0.50:8086, ] ] log_flow = 0 mode = VROUTER platform = HOST vr_limits= [ max_labels = 5120 max_nexthops = 524288 max_interfaces = 4355 max_vrfs = 4096 max_mirror_entries = 255 vrouter_max_bridge_entries = 262144 vrouter_max_oflow_bridge_entries = 53248 vrouter_build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-09-01 00:11:15.602301", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-10717-1.novalocal"}]} vrouter_max_flow_entries = 524288 vrouter_max_oflow_entries = 105472 vrouter_priority_tagging = 1 ] bgpaas_enabled = 0 port_mirror_enabled = 0 subcluster_name = vr_high_watermark = 80 vr_low_watermark = 75 vr_limit_exceeded_list= [ [ ] ] res_limit = 0 res_table_limit = 0 loopback_ip = 0.0.0.0 gateway_list= [ [ (*_iter161) = 10.0.0.1, ] ] ] 2025-09-01 Mon 01:37:55:394.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 01:37:55:394.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:395.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690675394478 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 279 in_bytes = 43179 out_tpkts = 245 out_bytes = 81747 exception_packets = 20 exception_packets_dropped = 11 exception_packets_allowed = 9 total_flows = 0 aged_flows = 0 uptime = 1756690644892846 phy_if_info= [ [ _iter254->first = ens3 [ speed = -1 duplexity = 255 ], ] ] raw_vhost_stats= [ name = vhost0 in_pkts = 279 in_bytes = 43179 out_pkts = 245 out_bytes = 81747 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 228 in_bytes = 81033 out_pkts = 279 out_bytes = 42976 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:37:55:395.475 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:37:55:395.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:395.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690675394672 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:37:55:395.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:37:55:811.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:811.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690675811049 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:55:811.242 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:55:812.061 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:812.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690675812048 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38025 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:55:812.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:55:812.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:55:812.300 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:55:812.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:812.391 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690675812322 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:55:812.397 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:55:812.404 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:55:812.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690675812330 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:55:812.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:57:403.188 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:37:57:404.439 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:37:57:404.609 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:37:57:404.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:37:57:404.643 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:37:57:404.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690677403280 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:57:404.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:57:404.692 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:37:57:404.697 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:37:57:404.732 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:57:404.757 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:37:57:404.798 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:57:404.909 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690677404547 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:57:404.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:57:404.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:57:404.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690677404716 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:37:57:404.950 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:37:57:404.956 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:37:57:404.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:57:405.092 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690677404789 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 30, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:37:57:405.107 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:37:58:718.969 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:58:719.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690678718853 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:58:719.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:58:719.874 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:58:719.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690678719860 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44315 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:58:719.917 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:58:720.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:37:58:720.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:37:58:720.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:58:720.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690678720139 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:58:720.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:37:58:720.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:37:58:720.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690678720145 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:37:58:720.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:23:352.441 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:23:352.516 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690703352407 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:23:352.528 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:23:352.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:23:352.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690703352792 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42129 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:23:352.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:23:353.082 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:38:23:353.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:23:353.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:23:353.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690703353217 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:23:353.258 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:23:353.268 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:23:353.275 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690703353224 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:23:353.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:25:385.114 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:385.205 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690705384942 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 88 dequeues = 88 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 107 dequeues = 107 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690677404756 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690703353267 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:38:25:385.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:25:385.248 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:385.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690705385064 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 34, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:25:385.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:25:394.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:394.867 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690705394779 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.285 five_min_avg = 0.24 fifteen_min_avg = 0.16 ] cpu_share = 1.32935 ] ] 2025-09-01 Mon 01:38:25:394.875 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:38:25:394.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:394.896 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690705394799 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.32935 used_sys_mem = 2523939144 one_min_cpuload = 0.285 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:38:25:394.902 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:38:25:394.909 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:395.017 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690705394826 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 441 in_bytes = 60395 out_tpkts = 390 out_bytes = 107605 exception_packets = 21 exception_packets_allowed = 10 raw_vhost_stats= [ name = vhost0 in_pkts = 441 in_bytes = 60395 out_pkts = 390 out_bytes = 107605 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 358 in_bytes = 106261 out_pkts = 441 out_bytes = 60192 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:38:25:395.039 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:38:25:395.046 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:25:395.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690705394851 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:38:25:395.072 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:38:27:404.832 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:38:27:405.098 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:38:27:405.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:38:27:405.234 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690707404917 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:38:27:405.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:38:27:405.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:38:27:405.275 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:38:27:405.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:38:27:405.401 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690707405188 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 34, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:27:405.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:27:405.423 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:38:27:405.428 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:38:27:405.458 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:27:405.481 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Idle 2025-09-01 Mon 01:38:27:405.518 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:27:405.531 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690707405443 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:38:27:405.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:38:27:405.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:38:27:405.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:27:405.687 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690707405509 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 35, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 96, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:27:405.702 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:29:230.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:29:231.077 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:38:29:233.014 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690709230255 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:29:233.023 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:29:233.032 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:29:233.043 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690709230815 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38567 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:29:233.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:29:233.068 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:29:233.207 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:29:233.238 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690709233198 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:29:233.257 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:29:233.264 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:29:233.272 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690709233205 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:29:233.277 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:51:240.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42632 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:38:51:245.071 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42638 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:38:51:703.409 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:51:703.476 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690731703372 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:51:703.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:51:703.910 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:51:703.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690731703884 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37815 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:51:703.930 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:51:704.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:38:51:704.181 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:51:704.253 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:51:704.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690731704217 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:51:704.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:51:704.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:51:704.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690731704231 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:51:704.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:55:385.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:55:385.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690735385162 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 106 dequeues = 106 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 128 dequeues = 128 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690707405480 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690731704315 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:38:55:385.411 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:55:385.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:55:385.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690735385286 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 41, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 106, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:55:385.548 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:55:395.044 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:55:395.212 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690735395010 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 617 in_bytes = 87811 out_tpkts = 555 out_bytes = 134139 exception_packets = 22 exception_packets_dropped = 12 raw_vhost_stats= [ name = vhost0 in_pkts = 617 in_bytes = 87811 out_pkts = 555 out_bytes = 134139 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.6955e-14 total_out_bandwidth_utilization = 3.2255e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6817, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5950, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 508 in_bytes = 132165 out_pkts = 617 out_bytes = 87608 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:38:55:395.232 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:38:55:395.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:55:395.275 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690735395050 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:38:55:395.281 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:38:57:043.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:043.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690737043294 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:57:043.402 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:57:043.740 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:043.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690737043732 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42567 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:57:043.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:57:043.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:38:57:044.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:57:044.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:044.079 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690737044044 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:57:044.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:57:044.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:044.113 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690737044052 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:38:57:044.120 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:38:57:405.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:38:57:405.943 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:38:57:406.094 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:38:57:406.127 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690737405707 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:38:57:406.138 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:38:57:406.148 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:38:57:406.165 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:38:57:406.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:38:57:406.349 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690737406059 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 114, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:57:406.378 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:38:57:406.390 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:38:57:406.398 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:38:57:406.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:38:57:406.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:38:57:406.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:406.574 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690737406424 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:38:57:406.583 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:38:57:406.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:38:57:406.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:38:57:406.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690737406539 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 26 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 116, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:38:57:406.830 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:20:864.344 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:20:864.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690760864305 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:20:864.425 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:20:864.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:20:864.819 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690760864793 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34185 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:20:864.842 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:20:865.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:39:20:865.126 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:20:865.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:20:865.191 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690760865157 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:20:865.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:20:865.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:20:865.211 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690760865163 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:20:865.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:25:385.534 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:385.648 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690765385367 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 122 dequeues = 122 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 147 dequeues = 147 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690737406486 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690760865202 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:39:25:385.674 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:25:385.685 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:385.896 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690765385503 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 46, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 122, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:25:385.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:25:395.586 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:395.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690765395557 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.185 five_min_avg = 0.22 fifteen_min_avg = 0.16 ] cpu_share = 1.18333 ] ] 2025-09-01 Mon 01:39:25:395.664 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:39:25:395.672 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:395.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690765395574 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.18333 used_sys_mem = 2523939144 one_min_cpuload = 0.185 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:39:25:395.690 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:39:25:395.696 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:395.810 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690765395621 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 1440 in_bytes = 266277 out_tpkts = 1306 out_bytes = 304609 raw_vhost_stats= [ name = vhost0 in_pkts = 1440 in_bytes = 266277 out_pkts = 1306 out_bytes = 304609 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 1244 in_bytes = 302005 out_pkts = 1440 out_bytes = 266074 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:39:25:395.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:39:25:395.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:25:395.877 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690765395653 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:39:25:395.885 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:39:26:204.216 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:26:204.284 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690766204178 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:26:204.294 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:26:204.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:26:204.384 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690766204357 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 46363 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:26:204.390 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:26:204.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:39:26:204.644 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:26:204.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:26:204.696 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690766204670 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:26:204.702 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:26:204.708 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:26:204.715 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690766204679 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:26:204.721 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:27:406.584 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:39:27:406.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:39:27:406.956 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:39:27:406.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690767406676 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:39:27:407.004 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:39:27:407.046 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:39:27:407.021 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:39:27:407.139 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:39:27:407.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690767406939 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 68 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 48, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 132, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:27:407.282 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:27:407.291 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:39:27:407.296 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:39:27:407.336 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:27:407.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:39:27:407.428 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:27:407.443 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690767407316 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:39:27:407.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:39:27:407.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:39:27:407.484 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:27:407.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690767407418 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 29 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 68 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 49, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 134, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:27:407.593 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:51:307.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:60036 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:39:51:312.574 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:60038 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:39:51:645.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:51:645.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690791645273 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:51:645.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:51:645.455 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:51:645.468 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690791645449 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41729 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:51:645.474 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:51:645.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:39:51:645.724 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:51:645.765 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:51:645.790 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690791645754 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:51:645.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:51:645.804 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:51:645.812 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690791645761 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:51:645.817 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:55:385.821 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:385.900 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690795385675 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 140 dequeues = 140 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 168 dequeues = 168 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690767407368 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690791645803 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:39:55:385.913 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:55:385.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:386.047 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690795385781 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 32 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 72 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 140, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:55:386.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:55:395.824 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:396.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690795395790 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 1622 in_bytes = 294510 out_tpkts = 1482 out_bytes = 331657 exception_packets = 23 exception_packets_dropped = 13 raw_vhost_stats= [ name = vhost0 in_pkts = 1622 in_bytes = 294510 out_pkts = 1482 out_bytes = 331657 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 1.41852e-13 total_out_bandwidth_utilization = 1.49398e-13 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 26167, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 27559, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 1405 in_bytes = 328423 out_pkts = 1622 out_bytes = 294307 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:39:55:396.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:39:55:396.050 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:396.071 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690795395833 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:39:55:396.076 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:39:55:904.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:904.916 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690795904823 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:55:904.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:55:905.321 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:905.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690795905287 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36961 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:55:905.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:55:905.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:39:55:905.547 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:55:905.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:905.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690795905577 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:55:905.637 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:55:905.643 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:55:905.650 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690795905584 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:39:55:905.655 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:39:57:407.441 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:39:57:408.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:39:57:408.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:39:57:408.190 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690797407523 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:39:57:408.198 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:39:57:408.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:39:57:408.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:39:57:408.313 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:39:57:408.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690797408150 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 76 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 148, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:57:408.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:39:57:408.359 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:39:57:408.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:39:57:408.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:39:57:408.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 01:39:57:408.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:57:408.514 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690797408411 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:39:57:408.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:39:57:408.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:39:57:408.562 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:39:57:408.654 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690797408490 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 34 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 76 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 150, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:39:57:408.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:19:995.932 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:19:996.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:40:19:997.769 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690819995904 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:19:997.779 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:19:997.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:19:997.816 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690819996538 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33665 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:19:997.968 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:19:997.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:19:998.037 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:19:998.079 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690819998024 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:19:998.088 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:19:998.097 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:19:998.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690819998031 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:19:998.114 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:25:386.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:386.182 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690825385933 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 156 dequeues = 156 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 187 dequeues = 187 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690797408456 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690819998096 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:40:25:386.195 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:25:386.216 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:386.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690825386056 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 35 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 58, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 156, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:25:386.341 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:25:396.024 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:396.103 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690825395996 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.065 five_min_avg = 0.18 fifteen_min_avg = 0.145 ] cpu_share = 1.175 ] ] 2025-09-01 Mon 01:40:25:396.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:40:25:396.123 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:396.135 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690825396011 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.175 used_sys_mem = 2523939144 one_min_cpuload = 0.065 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:40:25:396.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:40:25:396.146 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:396.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690825396043 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 1771 in_bytes = 310823 out_tpkts = 1621 out_bytes = 355366 exception_packets = 24 exception_packets_allowed = 11 raw_vhost_stats= [ name = vhost0 in_pkts = 1771 in_bytes = 310823 out_pkts = 1621 out_bytes = 355366 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 1529 in_bytes = 351502 out_pkts = 1771 out_bytes = 310620 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:40:25:396.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:40:25:396.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:25:396.305 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690825396066 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:40:25:396.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:40:26:956.051 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:26:956.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690826955999 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:26:956.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:26:956.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:26:956.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690826956292 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 39107 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:26:956.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:26:956.539 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:40:26:956.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:26:956.624 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:26:956.634 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690826956615 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:26:956.644 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:26:956.650 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:26:956.657 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690826956622 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:26:956.661 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:27:408.552 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:40:27:408.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:40:27:408.948 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:40:27:409.031 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690827408640 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:40:27:409.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:40:27:409.090 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:40:27:409.059 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:40:27:409.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:40:27:409.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690827408931 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 84 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 60, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 166, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:27:409.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:27:409.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:40:27:409.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:40:27:409.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:27:409.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 01:40:27:409.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:27:409.366 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690827409261 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:40:27:409.375 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:40:27:409.380 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:40:27:409.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:27:409.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690827409343 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 37 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 84 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 61, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 168, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:27:409.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:49:158.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:49:159.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690849158151 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:49:159.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:49:158.813 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:40:49:159.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:49:159.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690849158613 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37211 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:49:159.806 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:49:159.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:49:159.891 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:49:159.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690849159882 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:49:159.942 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:49:159.947 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:49:159.955 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690849159887 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:49:159.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:51:331.645 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:44208 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:40:51:336.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:44222 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:40:55:036.791 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:036.867 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690855036742 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:55:036.879 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:55:037.198 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:037.222 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690855037186 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44367 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:55:037.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:55:037.425 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:40:55:037.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:55:037.562 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:037.640 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690855037551 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:55:037.649 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:55:037.667 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:037.676 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690855037558 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:40:55:037.681 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:40:55:386.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:386.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690855386167 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 178 dequeues = 178 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 212 dequeues = 212 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690827409306 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690855037666 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:40:55:386.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:55:386.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:386.477 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690855386261 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 92 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 67, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 178, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:55:386.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:55:396.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:396.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690855396200 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 1943 in_bytes = 338524 out_tpkts = 1785 out_bytes = 381695 raw_vhost_stats= [ name = vhost0 in_pkts = 1943 in_bytes = 338524 out_pkts = 1785 out_bytes = 381695 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.52528e-14 total_out_bandwidth_utilization = 3.18105e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6503, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5868, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 1678 in_bytes = 377201 out_pkts = 1943 out_bytes = 338321 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:40:55:396.380 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:40:55:396.388 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:55:396.411 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690855396229 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:40:55:396.416 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:40:57:410.493 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:40:57:411.588 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:40:57:411.696 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:40:57:411.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690857410566 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:40:57:411.731 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:40:57:411.739 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:40:57:411.845 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690857411680 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 92 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 67, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 182, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:57:411.872 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:40:57:411.852 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:40:57:411.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:40:57:411.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:40:57:411.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:40:57:412.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:40:57:412.458 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 01:40:57:412.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:57:412.575 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690857412001 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:40:57:412.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:40:57:412.588 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:40:57:412.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:40:57:412.724 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690857412541 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 92 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 68, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 184, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:40:57:412.744 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:16:430.047 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:16:430.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690876430013 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:16:430.120 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:16:430.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:16:430.462 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690876430441 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43917 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:16:430.468 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:16:430.701 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:41:16:430.784 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:16:430.826 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:16:430.844 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690876430815 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:16:430.850 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:16:430.856 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:16:430.864 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690876430821 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:16:430.869 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:20:519.622 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.22:34946 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:41:25:386.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:386.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690885386362 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 190 dequeues = 190 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 227 dequeues = 227 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690857412457 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690876430856 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:41:25:386.584 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:25:386.591 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:386.693 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690885386457 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 96 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 71, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 190, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:25:386.717 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:25:396.355 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:396.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690885396324 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.235 five_min_avg = 0.205 fifteen_min_avg = 0.155 ] cpu_share = 1.18333 ] ] 2025-09-01 Mon 01:41:25:396.422 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:41:25:396.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:396.446 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690885396337 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.18333 used_sys_mem = 2523939144 one_min_cpuload = 0.235 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:41:25:396.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:41:25:396.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:396.566 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690885396361 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2171 in_bytes = 380017 out_tpkts = 1988 out_bytes = 417782 exception_packets = 25 exception_packets_allowed = 12 raw_vhost_stats= [ name = vhost0 in_pkts = 2171 in_bytes = 380017 out_pkts = 1988 out_bytes = 417782 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 1866 in_bytes = 412658 out_pkts = 2171 out_bytes = 379814 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:41:25:396.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:41:25:396.601 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:25:396.620 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690885396384 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:41:25:396.628 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:41:26:357.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:26:357.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690886357689 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:26:357.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:26:358.123 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:26:358.163 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690886358094 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37679 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:26:358.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:26:358.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:41:26:358.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:26:358.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:26:358.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690886358401 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:26:358.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:26:358.446 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:26:358.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690886358405 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:26:358.464 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:27:412.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:41:27:413.125 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:41:27:413.253 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:41:27:413.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690887412666 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:41:27:413.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:41:27:413.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:41:27:413.338 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:41:27:413.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:41:27:413.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690887413235 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 73, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 200, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:27:413.422 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:27:413.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:41:27:413.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:41:27:413.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:27:413.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:41:27:413.533 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:27:413.546 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690887413452 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:41:27:413.555 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:41:27:413.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:41:27:413.585 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:27:413.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690887413524 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 74, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 202, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:27:413.730 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:44:510.991 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:44:511.303 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690904510952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:44:511.313 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:44:511.321 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:44:511.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690904511125 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42293 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:44:511.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:44:511.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:41:44:511.443 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:44:511.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:44:511.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690904511502 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:44:511.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:44:511.570 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:44:511.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690904511506 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:44:511.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:51:354.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48246 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:41:51:358.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48260 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:41:55:386.731 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:55:386.821 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690915386586 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 208 dequeues = 208 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 248 dequeues = 248 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690887413495 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690904511570 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:41:55:386.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:55:386.841 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:55:386.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690915386684 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 49 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 104 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 26 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 78, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 208, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:55:386.993 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:55:396.549 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:55:396.707 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690915396519 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2349 in_bytes = 408483 out_tpkts = 2159 out_bytes = 446038 exception_packets = 27 exception_packets_dropped = 14 exception_packets_allowed = 13 raw_vhost_stats= [ name = vhost0 in_pkts = 2349 in_bytes = 408483 out_pkts = 2159 out_bytes = 446038 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 4.55961e-14 total_out_bandwidth_utilization = 5.05618e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 8411, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 9327, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 2022 in_bytes = 440284 out_pkts = 2349 out_bytes = 408280 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:41:55:396.734 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:41:55:396.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:55:396.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690915396554 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:41:55:396.769 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:41:56:868.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:56:868.616 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690916868526 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:56:868.626 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:56:868.730 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:56:868.745 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690916868722 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 39381 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:56:868.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:56:869.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:41:56:869.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:56:869.164 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:56:869.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690916869139 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:56:869.212 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:56:869.221 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:56:869.228 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690916869144 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:41:56:869.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:41:57:413.562 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:41:57:414.221 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:41:57:414.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:41:57:414.354 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690917413672 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:41:57:414.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:41:57:414.370 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:41:57:414.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:41:57:414.554 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:41:57:414.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690917414310 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 50 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 108 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 80, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 216, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:57:414.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:41:57:414.601 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:41:57:414.606 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:41:57:414.653 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:41:57:414.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:41:57:414.732 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:57:414.748 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690917414630 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:41:57:414.754 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:41:57:414.759 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:41:57:414.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:41:57:414.896 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690917414722 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 108 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 81, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 218, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:41:57:414.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:16:641.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:16:641.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690936641647 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:16:641.793 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:16:642.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:16:642.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690936642144 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33469 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:16:642.230 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:16:642.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:42:16:642.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:16:642.499 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:16:642.510 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690936642490 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:16:642.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:16:642.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:16:642.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690936642497 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:16:642.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:25:386.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:387.051 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690945386797 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 224 dequeues = 224 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 267 dequeues = 267 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690917414682 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690936642526 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:42:25:387.076 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:25:387.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:387.254 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690945386914 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 52 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 112 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 28 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 224, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:25:387.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:25:396.709 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:396.787 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690945396674 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.085 five_min_avg = 0.165 fifteen_min_avg = 0.145 ] cpu_share = 1.15833 ] ] 2025-09-01 Mon 01:42:25:396.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:42:25:396.804 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:396.816 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690945396688 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.15833 used_sys_mem = 2523939144 one_min_cpuload = 0.085 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:42:25:396.823 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:42:25:396.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:396.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690945396714 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2501 in_bytes = 424317 out_tpkts = 2299 out_bytes = 469997 exception_packets = 28 exception_packets_allowed = 14 raw_vhost_stats= [ name = vhost0 in_pkts = 2501 in_bytes = 424317 out_pkts = 2299 out_bytes = 469997 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 2149 in_bytes = 463697 out_pkts = 2501 out_bytes = 424114 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:42:25:396.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:42:25:396.947 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:25:396.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756690945396736 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:42:25:396.974 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:42:26:029.272 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:26:029.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690946029239 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:26:029.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:26:029.677 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:26:029.710 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756690946029657 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40211 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:26:029.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:26:029.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:42:26:029.962 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:26:030.014 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:26:030.037 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690946030005 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:26:030.044 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:26:030.050 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:26:030.058 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690946030011 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:26:030.075 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:27:414.754 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:42:27:416.740 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:42:27:416.845 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:42:27:416.913 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690947416579 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:42:27:416.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:42:27:416.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:42:27:416.946 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:42:27:416.972 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:42:27:417.051 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690947416820 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 53 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 116 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 29 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 234, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:27:417.078 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:27:417.086 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:42:27:417.091 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:42:27:417.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:27:417.152 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Idle 2025-09-01 Mon 01:42:27:417.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:27:417.214 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690947417107 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:42:27:417.219 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:42:27:417.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:42:27:417.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:27:417.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690947417186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 116 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 29 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 236, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:27:417.388 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:46:612.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:46:612.783 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690966612651 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:46:612.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:46:613.318 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:46:613.360 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690966613293 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37005 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:46:613.368 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:46:613.511 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:42:46:613.571 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:46:613.680 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:46:613.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690966613619 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:46:613.707 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:46:613.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:46:613.728 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756690966613627 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:46:613.735 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:51:417.860 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:55456 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:42:51:422.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:55470 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:42:54:650.160 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:54:650.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690974650129 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:54:650.235 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:54:650.638 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:54:650.686 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690974650612 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45085 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:54:650.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:54:650.813 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:42:54:650.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:54:650.921 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:54:650.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690974650911 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:54:650.952 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:54:650.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:54:650.970 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756690974650918 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:42:54:650.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:42:55:387.210 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:55:387.288 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690975387049 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 246 dequeues = 246 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 292 dequeues = 292 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690947417152 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756690974650958 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:42:55:387.299 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:55:387.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:55:387.432 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690975387147 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 246, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:55:387.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:55:396.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:55:397.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690975396883 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2698 in_bytes = 461857 out_tpkts = 2480 out_bytes = 503814 exception_packets = 29 exception_packets_allowed = 15 raw_vhost_stats= [ name = vhost0 in_pkts = 2698 in_bytes = 461857 out_pkts = 2480 out_bytes = 503814 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 11102 out_bandwidth_usage = 11165 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 4.12702e-14 total_out_bandwidth_utilization = 3.86084e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 7613, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 7122, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 2327 in_bytes = 497388 out_pkts = 2699 out_bytes = 461696 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:42:55:397.077 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:42:55:397.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:55:397.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690975396915 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:42:55:397.110 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:42:57:417.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:42:57:417.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:42:57:418.084 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:42:57:418.989 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:42:57:419.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:42:57:419.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690977417309 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:42:57:419.123 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:42:57:419.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:42:57:419.150 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:42:57:419.183 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:42:57:419.208 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:42:57:419.246 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:57:419.349 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690977419059 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 58 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 250, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:57:419.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:42:57:419.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:57:419.384 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690977419167 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:42:57:419.389 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:42:57:419.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:42:57:419.416 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:42:57:419.503 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690977419237 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 251, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:42:57:419.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:15:233.745 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:15:234.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756690995233718 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:15:234.082 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:15:234.107 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:15:234.120 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756690995234100 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40849 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:15:234.127 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:15:234.390 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:43:15:234.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:15:234.648 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:15:234.668 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690995234637 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:15:234.674 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:15:234.680 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:15:234.688 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756690995234642 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:15:234.692 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:23:271.088 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:23:271.155 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691003271050 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:23:271.182 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:23:271.589 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:23:271.633 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691003271566 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41409 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:23:271.642 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:23:271.787 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:43:23:271.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:23:271.863 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:23:271.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691003271854 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:23:271.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:23:271.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:23:271.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691003271860 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:23:271.931 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:25:387.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:387.471 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691005387259 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 262 dequeues = 262 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 311 dequeues = 311 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756690977419207 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691003271919 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:43:25:387.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:25:387.506 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:387.619 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691005387351 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 132 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 97, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 262, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:25:387.632 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:25:397.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:397.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691005397066 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.03 five_min_avg = 0.135 fifteen_min_avg = 0.135 ] cpu_share = 1.15 ] ] 2025-09-01 Mon 01:43:25:397.162 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:43:25:397.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:397.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691005397077 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.15 used_sys_mem = 2523939144 one_min_cpuload = 0.03 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:43:25:397.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:43:25:397.198 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:397.284 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691005397102 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2838 in_bytes = 477454 out_tpkts = 2605 out_bytes = 527376 raw_vhost_stats= [ name = vhost0 in_pkts = 2838 in_bytes = 477454 out_pkts = 2605 out_bytes = 527376 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 2449 in_bytes = 520824 out_pkts = 2839 out_bytes = 477293 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:43:25:397.309 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:43:25:397.316 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:25:397.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691005397124 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:43:25:397.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:43:27:419.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:43:27:419.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:43:27:419.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:43:27:419.948 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691007419368 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:43:27:419.956 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:43:27:419.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:43:27:420.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691007419910 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 132 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 97, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 268, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:27:420.148 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:27:420.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:43:27:420.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:43:27:420.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:43:27:420.234 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:43:27:420.275 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:27:420.308 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 01:43:27:420.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:27:420.380 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691007420254 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:43:27:420.386 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:43:27:420.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:43:27:420.422 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:27:420.517 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691007420353 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 62 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 132 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 33 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 270, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:27:420.533 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:44:394.745 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:44:394.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691024394711 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:44:394.817 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:44:395.162 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:44:395.177 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691024395153 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38237 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:44:395.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:44:395.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:43:44:395.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:44:395.715 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:44:395.731 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691024395707 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:44:395.736 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:44:395.741 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:44:395.748 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691024395712 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:44:395.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:51:082.022 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:51:082.094 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691031081988 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:51:082.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:51:082.416 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:51:082.432 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691031082408 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37675 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:51:082.437 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:51:082.614 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:43:51:082.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:51:082.760 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:51:082.779 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691031082750 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:51:082.784 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:51:082.796 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:51:082.803 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691031082758 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:43:51:082.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:43:51:441.170 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42702 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:43:51:445.479 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42706 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:43:55:387.596 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:55:387.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691035387457 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 280 dequeues = 280 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 332 dequeues = 332 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691007420307 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691031082795 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:43:55:387.710 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:55:387.717 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:55:387.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691035387567 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 66 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 140 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 35 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 280, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:55:387.858 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:55:397.301 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:55:397.448 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691035397272 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 2995 in_bytes = 504143 out_tpkts = 2752 out_bytes = 553126 raw_vhost_stats= [ name = vhost0 in_pkts = 2995 in_bytes = 504143 out_pkts = 2752 out_bytes = 553126 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.54588e-14 total_out_bandwidth_utilization = 3.05637e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6541, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5638, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 2593 in_bytes = 546448 out_pkts = 2996 out_bytes = 503982 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:43:55:397.471 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:43:55:397.479 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:55:397.499 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691035397302 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:43:55:397.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:43:57:420.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:43:57:421.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:43:57:421.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:43:57:421.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691037420511 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:43:57:421.226 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:43:57:421.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:43:57:421.254 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:43:57:421.292 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:43:57:421.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691037421130 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 62 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 66 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 140 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 35 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 284, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:57:421.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:43:57:421.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:43:57:421.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:43:57:421.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:43:57:421.462 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:43:57:421.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:57:421.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691037421411 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:43:57:421.526 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:43:57:421.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:43:57:421.574 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:43:57:421.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691037421495 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 140 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 35 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 105, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 286, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:43:57:421.739 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:14:095.926 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:14:096.006 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691054095894 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:14:096.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:14:096.398 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:44:14:096.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:14:096.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691054096102 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41957 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:14:096.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:14:096.522 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:14:096.566 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:14:096.602 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691054096549 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:14:096.609 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:14:096.616 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:14:096.626 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691054096554 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:14:096.632 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:18:892.913 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:18:892.997 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691058892879 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:18:893.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:18:893.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:18:893.248 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691058893083 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35483 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:18:893.253 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:18:893.355 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:44:18:893.398 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:18:893.452 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:18:893.472 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691058893443 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:18:893.478 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:18:893.484 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:18:893.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691058893448 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:18:893.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:25:387.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:387.936 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691065387697 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 296 dequeues = 296 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 351 dequeues = 351 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691037421461 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691058893483 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:44:25:387.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:25:387.987 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:388.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691065387812 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 69 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 148 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 37 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 109, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 296, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:25:388.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:25:397.468 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:397.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691065397426 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.07 five_min_avg = 0.125 fifteen_min_avg = 0.13 ] cpu_share = 1.15833 ] ] 2025-09-01 Mon 01:44:25:397.533 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:44:25:397.540 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:397.552 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691065397457 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.15833 used_sys_mem = 2523939144 one_min_cpuload = 0.07 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:44:25:397.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:44:25:397.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:397.658 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691065397482 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 3796 in_bytes = 680336 out_tpkts = 3484 out_bytes = 722889 raw_vhost_stats= [ name = vhost0 in_pkts = 3796 in_bytes = 680336 out_pkts = 3484 out_bytes = 722889 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 3322 in_bytes = 716085 out_pkts = 3797 out_bytes = 680175 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:44:25:397.676 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:44:25:397.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:25:397.702 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691065397522 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:44:25:397.710 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:44:27:421.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:44:27:422.049 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:44:27:422.164 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:44:27:422.198 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691067421670 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:44:27:422.207 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:44:27:422.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:44:27:422.320 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:44:27:422.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:44:27:422.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691067422150 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 66 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 69 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 148 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 37 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 109, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 302, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:27:422.388 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:27:422.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:44:27:422.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:44:27:422.445 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:27:422.472 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:44:27:422.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:27:422.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691067422428 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:44:27:422.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:44:27:422.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:44:27:422.568 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:27:422.662 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691067422505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 67 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 70 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 148 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 37 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 110, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 304, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:27:422.676 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:42:716.720 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:42:716.780 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691082716688 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:42:716.790 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:42:716.902 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:42:716.916 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691082716894 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40795 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:42:716.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:42:717.156 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:44:42:717.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:42:717.276 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:42:717.325 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691082717254 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:42:717.330 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:42:717.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:42:717.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691082717268 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:42:717.347 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:49:943.644 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:49:943.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691089943602 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:49:943.721 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:49:944.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:49:944.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691089944127 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45501 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:49:944.200 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:49:944.384 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:44:49:944.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:49:944.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:49:944.513 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691089944489 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:49:944.524 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:49:944.531 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:49:944.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691089944496 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:44:49:944.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:44:51:464.041 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48424 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:44:51:469.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48426 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:44:55:388.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:55:388.194 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691095387969 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 314 dequeues = 314 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 372 dequeues = 372 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691067422471 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691089944530 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:44:55:388.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:55:388.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:55:388.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691095388084 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 68 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 74 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 156 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 39 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 116, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 314, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:55:388.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:55:397.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:55:397.904 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691095397698 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 3964 in_bytes = 707868 out_tpkts = 3646 out_bytes = 750177 exception_packets = 30 exception_packets_allowed = 16 raw_vhost_stats= [ name = vhost0 in_pkts = 3964 in_bytes = 707868 out_pkts = 3646 out_bytes = 750177 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 1.42242e-13 total_out_bandwidth_utilization = 1.47251e-13 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 26239, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 27163, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 3481 in_bytes = 743247 out_pkts = 3965 out_bytes = 707707 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:44:55:397.936 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:44:55:397.950 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:55:397.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691095397740 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:44:55:397.998 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:44:57:422.534 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:44:57:423.250 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:44:57:423.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:44:57:423.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691097422636 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:44:57:423.402 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:44:57:423.458 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:44:57:423.462 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:44:57:423.493 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:44:57:423.599 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691097423347 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 70 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 74 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 156 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 39 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 116, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 318, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:57:423.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:44:57:423.635 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:44:57:423.641 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:44:57:423.675 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:44:57:423.709 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:44:57:423.760 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:57:423.780 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691097423658 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:44:57:423.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:44:57:423.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:44:57:423.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:44:57:423.942 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691097423751 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 71 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 156 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 39 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 117, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 320, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:44:57:423.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:10:797.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:10:797.547 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691110797419 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:10:797.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:10:797.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:10:797.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691110797913 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34447 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:10:797.957 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:10:798.142 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:45:10:798.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:10:798.281 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:10:798.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691110798270 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:10:798.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:10:798.320 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:10:798.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691110798278 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:10:798.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:20:724.680 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:20:724.866 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691120724644 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:20:724.882 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:20:724.892 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:20:724.906 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691120724810 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 46481 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:20:724.913 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:20:725.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:45:20:725.097 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:20:725.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:20:725.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691120725141 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:20:725.228 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:20:725.236 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:20:725.246 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691120725146 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:20:725.254 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:25:388.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:388.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691125388178 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 330 dequeues = 330 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 19 dequeues = 19 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 391 dequeues = 391 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691097423708 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691120725235 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:45:25:388.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:25:388.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:388.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691125388287 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 72 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 77 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 164 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 121, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 330, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:25:388.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:25:397.910 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:398.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691125397878 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.025 five_min_avg = 0.1 fifteen_min_avg = 0.125 ] cpu_share = 1.14148 ] ] 2025-09-01 Mon 01:45:25:398.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:45:25:398.028 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:398.042 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691125397892 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.14148 used_sys_mem = 2523939144 one_min_cpuload = 0.025 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:45:25:398.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:45:25:398.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:398.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691125397915 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4119 in_bytes = 725891 out_tpkts = 3792 out_bytes = 774911 exception_packets = 31 exception_packets_allowed = 17 raw_vhost_stats= [ name = vhost0 in_pkts = 4119 in_bytes = 725891 out_pkts = 3792 out_bytes = 774911 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 3624 in_bytes = 767855 out_pkts = 4120 out_bytes = 725730 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:45:25:398.195 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:45:25:398.201 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:25:398.221 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691125397957 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:45:25:398.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:45:27:423.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:45:27:424.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:45:27:424.341 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:45:27:424.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691127423877 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:45:27:424.371 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:45:27:424.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:45:27:424.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691127424318 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 74 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 77 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 164 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 121, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 336, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:27:424.528 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:27:424.529 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:45:27:424.573 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:45:27:424.619 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:45:27:424.629 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:45:27:424.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:27:424.837 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:45:27:424.891 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:27:424.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691127424649 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:45:27:424.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:45:27:424.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:45:27:424.951 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:27:425.074 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691127424881 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 75 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 164 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 122, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 338, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:27:425.090 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:38:608.440 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:38:608.812 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:45:38:608.848 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691138608407 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:38:608.877 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:38:608.888 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:38:608.902 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691138608597 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34857 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:38:608.909 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:38:608.942 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:38:609.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:38:609.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691138608997 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:38:609.250 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:38:609.258 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:38:609.269 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691138609011 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:38:609.276 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:51:486.685 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:43672 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:45:51:490.927 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:43678 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:45:52:585.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:52:585.492 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691152585267 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:52:585.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:52:585.523 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:52:585.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691152585441 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37081 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:52:585.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:52:585.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:45:52:585.758 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:52:585.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:52:585.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691152585802 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:52:585.953 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:52:585.962 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:52:585.974 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691152585807 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:45:52:585.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:45:55:388.553 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:55:388.630 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691155388415 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 348 dequeues = 348 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 412 dequeues = 412 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691127424836 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691152585961 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:45:55:388.642 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:55:388.667 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:55:388.773 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691155388516 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 76 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 82 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 172 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 43 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 128, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 348, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:55:388.787 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:55:398.150 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:55:398.295 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691155398117 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4287 in_bytes = 754879 out_tpkts = 3953 out_bytes = 802709 exception_packets = 32 exception_packets_dropped = 15 raw_vhost_stats= [ name = vhost0 in_pkts = 4287 in_bytes = 754879 out_pkts = 3953 out_bytes = 802709 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.77844e-14 total_out_bandwidth_utilization = 3.39789e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6970, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 6268, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 3782 in_bytes = 795527 out_pkts = 4288 out_bytes = 754718 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:45:55:398.318 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:45:55:398.325 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:55:398.344 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691155398160 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:45:55:398.350 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:45:57:424.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:45:57:426.824 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:45:57:426.946 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:45:57:426.971 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691157426634 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:45:57:426.999 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:45:57:427.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:45:57:427.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:45:57:427.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:45:57:427.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691157426916 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 78 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 82 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 172 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 43 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 128, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 352, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:57:427.197 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:45:57:427.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:45:57:427.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:45:57:427.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:45:57:427.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:45:57:427.332 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:57:427.345 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691157427240 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:45:57:427.351 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:45:57:427.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:45:57:427.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:45:57:427.475 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691157427322 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 83 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 172 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 43 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 129, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 354, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:45:57:427.488 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:10:469.187 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:10:469.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691170469152 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:10:469.266 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:10:469.535 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:10:469.548 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691170469527 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38195 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:10:469.554 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:10:469.837 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:46:10:469.923 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:10:469.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:10:470.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691170469952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:10:470.014 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:10:470.021 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:10:470.031 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691170469957 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:10:470.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:21:476.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:21:476.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691181475904 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:21:476.080 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:21:476.338 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:21:476.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691181476324 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43595 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:21:476.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:21:476.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:46:21:476.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:21:476.843 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:21:476.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691181476791 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:21:476.866 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:21:476.874 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:21:476.882 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691181476798 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:21:476.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:22:352.133 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.22:45782 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:46:25:388.794 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:388.895 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691185388642 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 364 dequeues = 364 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 431 dequeues = 431 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691157427286 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691181476873 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:46:25:388.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:25:388.915 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:389.048 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691185388750 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 86 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 180 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 134, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 364, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:25:389.062 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:25:398.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:398.386 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691185398287 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.44 five_min_avg = 0.19 fifteen_min_avg = 0.15 ] cpu_share = 1.16667 ] ] 2025-09-01 Mon 01:46:25:398.397 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:46:25:398.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:398.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691185398303 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.16667 used_sys_mem = 2523939144 one_min_cpuload = 0.44 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:46:25:398.435 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:46:25:398.442 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:398.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691185398332 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4504 in_bytes = 796019 out_tpkts = 4148 out_bytes = 838998 raw_vhost_stats= [ name = vhost0 in_pkts = 4504 in_bytes = 796019 out_pkts = 4148 out_bytes = 838998 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 3974 in_bytes = 831690 out_pkts = 4505 out_bytes = 795858 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:46:25:398.552 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:46:25:398.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:25:398.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691185398356 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:46:25:398.583 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:46:27:427.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:46:27:427.621 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:46:27:427.757 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:46:27:427.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691187427470 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:46:27:427.804 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:46:27:427.814 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:46:27:427.939 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:46:27:427.962 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691187427731 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 82 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 86 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 180 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 134, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 370, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:27:428.005 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:27:427.964 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:46:27:428.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:46:27:428.054 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:46:27:428.098 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:27:428.130 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Idle 2025-09-01 Mon 01:46:27:428.207 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:27:428.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691187428078 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:46:27:428.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:46:27:428.237 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:46:27:428.262 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:27:428.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691187428186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 83 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 180 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 45 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 135, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 372, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:27:428.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:42:600.149 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:42:600.230 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691202600110 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:42:600.257 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:42:600.748 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:42:600.785 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691202600722 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44707 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:42:600.793 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:42:601.014 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:46:42:601.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:42:601.253 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:42:601.283 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691202601220 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:42:601.293 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:42:601.302 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:42:601.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691202601234 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:42:601.320 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:48:746.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:48:747.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691208746923 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:48:747.043 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:48:747.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:48:747.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691208747520 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40971 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:48:747.601 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:48:747.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:46:48:747.736 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:48:747.780 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:48:747.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691208747768 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:48:747.805 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:48:747.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:48:747.818 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691208747778 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:46:48:747.822 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:46:51:507.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:39912 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:46:51:512.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:39920 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:46:55:389.030 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:55:389.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691215388876 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 382 dequeues = 382 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 452 dequeues = 452 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691187428129 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691208747810 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:46:55:389.133 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:55:389.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:55:389.311 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691215389004 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 84 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 91 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 188 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 141, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 382, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:55:389.326 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:55:398.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:55:398.697 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691215398505 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4670 in_bytes = 823180 out_tpkts = 4305 out_bytes = 865172 exception_packets = 34 exception_packets_allowed = 19 raw_vhost_stats= [ name = vhost0 in_pkts = 4670 in_bytes = 823180 out_pkts = 4305 out_bytes = 865172 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 4.49619e-14 total_out_bandwidth_utilization = 4.93963e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 8294, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 9112, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 4128 in_bytes = 857738 out_pkts = 4672 out_bytes = 823061 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:46:55:398.729 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:46:55:398.740 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:55:398.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691215398536 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:46:55:398.768 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:46:57:428.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:46:57:428.691 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:46:57:428.875 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:46:57:428.908 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:46:57:428.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:46:57:428.971 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691217428405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:46:57:428.981 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:46:57:428.998 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:46:57:429.181 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691217428853 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 86 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 91 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 188 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 141, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 386, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:57:429.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:46:57:429.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:46:57:429.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:46:57:429.272 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:46:57:429.316 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:46:57:429.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:57:429.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691217429248 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:46:57:429.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:46:57:429.416 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:46:57:429.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:46:57:429.649 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691217429366 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 87 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 92 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 188 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 47 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 142, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 388, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:46:57:429.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:12:841.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:12:841.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691232841402 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:12:841.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:12:841.930 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:12:841.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691232841903 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41029 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:12:842.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:12:842.201 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:47:12:842.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:12:842.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:12:842.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691232842275 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:12:842.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:12:842.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:12:842.405 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691232842286 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:12:842.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:20:337.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:20:338.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691240337879 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:20:338.011 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:20:338.338 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:20:338.355 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691240338330 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35321 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:20:338.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:20:338.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:47:20:338.687 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:20:338.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:20:338.749 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691240338715 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:20:338.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:20:338.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:20:338.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691240338722 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:20:338.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:25:389.243 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:390.616 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691245389081 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 398 dequeues = 398 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 471 dequeues = 471 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691217429315 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691240338762 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:47:25:390.635 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:25:390.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:390.756 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691245389175 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 88 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 94 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 196 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 49 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 146, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 398, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:25:390.778 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:25:398.697 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:398.778 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691245398670 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.16 five_min_avg = 0.155 fifteen_min_avg = 0.14 ] cpu_share = 1.14167 ] ] 2025-09-01 Mon 01:47:25:398.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:47:25:398.798 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:398.810 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691245398683 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.14167 used_sys_mem = 2523939144 one_min_cpuload = 0.16 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:47:25:398.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:47:25:398.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:398.926 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691245398713 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4810 in_bytes = 838672 out_tpkts = 4433 out_bytes = 888747 raw_vhost_stats= [ name = vhost0 in_pkts = 4810 in_bytes = 838672 out_pkts = 4433 out_bytes = 888747 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 4253 in_bytes = 881187 out_pkts = 4812 out_bytes = 838553 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:47:25:398.939 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:47:25:398.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:25:398.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691245398757 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:47:25:398.970 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:47:27:429.415 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:47:27:430.006 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:47:27:430.126 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:47:27:430.153 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691247429514 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:47:27:430.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:47:27:430.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:47:27:430.173 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:47:27:430.200 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:47:27:430.325 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691247430108 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 90 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 94 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 196 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 49 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 146, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 404, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:27:430.363 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:27:430.373 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:47:27:430.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:47:27:430.415 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:27:430.442 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Idle 2025-09-01 Mon 01:47:27:430.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:27:430.503 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691247430396 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:47:27:430.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:47:27:430.514 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:47:27:430.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:27:430.662 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691247430480 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 91 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 95 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 196 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 49 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 147, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 406, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:27:430.677 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:43:892.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:43:892.765 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:47:43:894.312 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691263892381 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:43:894.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:43:894.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:43:894.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:43:894.471 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691263892557 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44097 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:43:894.477 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:43:894.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:43:894.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691263894368 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:43:894.495 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:43:894.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:43:894.518 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691263894374 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:43:894.523 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:50:848.905 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:50:848.968 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691270848878 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:50:848.984 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:50:849.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:50:849.267 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691270849239 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36187 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:50:849.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:50:849.489 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:47:50:849.567 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:50:849.626 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:50:849.650 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691270849612 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:50:849.657 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:50:849.663 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:50:849.671 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691270849618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:47:50:849.676 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:47:51:574.622 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:45972 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:47:51:578.923 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:45974 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:47:55:389.455 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:55:389.526 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691275389322 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 416 dequeues = 416 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 492 dequeues = 492 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691247430442 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691270849663 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:47:55:389.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:55:389.563 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:55:389.694 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691275389424 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 92 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 204 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 416, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:55:389.709 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:55:398.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:55:399.065 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691275398861 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 4980 in_bytes = 866669 out_tpkts = 4598 out_bytes = 916710 exception_packets = 36 exception_packets_allowed = 21 raw_vhost_stats= [ name = vhost0 in_pkts = 4980 in_bytes = 866669 out_pkts = 4598 out_bytes = 916710 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 10976 out_bandwidth_usage = 10797 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.70689e-14 total_out_bandwidth_utilization = 3.14635e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6838, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5804, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 4415 in_bytes = 909024 out_pkts = 4983 out_bytes = 866592 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:47:55:399.092 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:47:55:399.101 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:55:399.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691275398892 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:47:55:399.127 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:47:57:430.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:47:57:431.072 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:47:57:431.202 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:47:57:431.227 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691277430608 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:47:57:431.234 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:47:57:431.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:47:57:431.257 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:47:57:431.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:47:57:431.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691277431165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 94 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 204 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 420, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:57:431.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:47:57:431.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:47:57:431.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:47:57:431.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:47:57:431.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 01:47:57:431.498 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:57:431.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691277431401 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:47:57:431.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:47:57:431.539 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:47:57:431.564 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:47:57:431.661 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691277431487 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 95 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 100 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 204 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 422, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:47:57:431.674 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:11:434.535 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:11:434.591 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691291434506 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:11:434.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:11:434.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:11:434.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691291434952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 39759 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:11:434.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:11:435.267 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:48:11:435.365 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:11:435.401 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:11:435.431 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691291435392 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:11:435.436 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:11:435.442 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:11:435.448 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691291435396 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:11:435.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:19:739.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:19:739.911 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691299739737 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:19:739.931 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:19:740.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:19:740.607 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691299740563 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 37817 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:19:740.616 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:19:740.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:48:19:741.058 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:19:741.155 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:19:741.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691299741092 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:19:741.179 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:19:741.188 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:19:741.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691299741098 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:19:741.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:25:389.679 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:389.757 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691305389537 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 432 dequeues = 432 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 26 dequeues = 26 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 511 dequeues = 511 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691277431452 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691299741186 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:48:25:389.769 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:25:389.776 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:389.879 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691305389647 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 96 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 102 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 212 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 53 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 158, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 432, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:25:389.904 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:25:399.098 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:399.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691305399064 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.215 five_min_avg = 0.16 fifteen_min_avg = 0.14 ] cpu_share = 0.925 ] ] 2025-09-01 Mon 01:48:25:399.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:48:25:399.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:399.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691305399081 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.925 used_sys_mem = 2523939144 one_min_cpuload = 0.215 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:48:25:399.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:48:25:399.220 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:399.317 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691305399115 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 5805 in_bytes = 1050200 out_tpkts = 5341 out_bytes = 1075422 raw_vhost_stats= [ name = vhost0 in_pkts = 5805 in_bytes = 1050200 out_pkts = 5341 out_bytes = 1075422 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5155 in_bytes = 1067610 out_pkts = 5808 out_bytes = 1050123 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:48:25:399.331 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:48:25:399.337 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:25:399.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691305399155 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:48:25:399.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:48:27:431.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:48:27:432.104 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:48:27:432.240 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:48:27:432.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691307431623 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:48:27:432.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:48:27:432.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:48:27:432.329 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:48:27:432.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:48:27:432.429 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691307432209 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 98 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 102 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 212 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 53 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 158, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 438, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:27:432.477 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:27:432.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:48:27:432.499 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:48:27:432.542 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:27:432.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:48:27:432.652 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:27:432.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691307432522 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:48:27:432.675 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:48:27:432.681 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:48:27:432.708 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:27:432.801 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691307432642 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 99 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 103 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 212 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 53 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 159, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 440, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:27:432.830 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:38:435.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:38:435.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691318435509 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:38:435.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:38:436.125 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:38:436.171 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691318436095 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40185 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:38:436.181 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:38:436.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:48:38:436.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:38:436.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:38:436.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691318436869 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:38:436.954 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:38:436.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:38:436.975 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691318436884 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:38:436.982 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:51:331.246 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:51:331.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691331331211 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:51:331.326 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:51:331.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:51:331.425 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691331331387 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35517 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:51:331.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:51:331.619 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:48:51:331.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:51:331.732 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:51:331.745 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691331331724 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:51:331.750 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:51:331.755 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:51:331.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691331331735 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:48:51:331.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:48:51:598.632 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:46086 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:48:51:603.087 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:46098 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:48:55:389.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:55:389.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691335389758 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 26 dequeues = 26 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 450 dequeues = 450 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 26 dequeues = 26 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 27 dequeues = 27 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 532 dequeues = 532 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691307432578 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691331331755 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:48:55:389.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:55:390.008 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:55:390.128 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691335389854 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 107 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 220 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 450, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:55:390.143 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:55:399.322 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:55:399.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691335399286 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 5976 in_bytes = 1078603 out_tpkts = 5512 out_bytes = 1108073 exception_packets = 37 exception_packets_allowed = 22 raw_vhost_stats= [ name = vhost0 in_pkts = 5976 in_bytes = 1078603 out_pkts = 5512 out_bytes = 1108073 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 1.38133e-13 total_out_bandwidth_utilization = 1.53182e-13 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 25481, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 28257, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5323 in_bytes = 1100135 out_pkts = 5979 out_bytes = 1078526 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:48:55:399.572 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:48:55:399.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:55:399.611 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691335399322 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:48:55:399.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:48:57:432.707 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:48:57:433.290 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:48:57:433.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:48:57:433.463 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:48:57:433.494 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:48:57:433.466 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691337432810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:48:57:433.518 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:48:57:433.526 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:48:57:433.649 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691337433409 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 102 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 107 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 220 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 454, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:57:433.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:48:57:433.678 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:48:57:433.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:48:57:433.728 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:48:57:433.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 01:48:57:433.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:57:433.812 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691337433702 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:48:57:433.823 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:48:57:433.832 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:48:57:433.858 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:48:57:433.952 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691337433786 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 103 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 220 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 456, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:48:57:433.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:08:137.043 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:08:137.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691348137005 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:08:137.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:08:137.345 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:08:137.358 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691348137335 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 42529 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:08:137.364 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:08:137.569 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:49:08:137.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:08:137.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:08:137.851 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691348137805 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:08:137.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:08:137.865 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:08:137.872 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691348137811 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:08:137.877 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:22:651.887 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:22:651.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691362651855 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:22:651.952 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:22:652.346 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:22:652.384 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691362652321 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35119 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:22:652.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:22:652.570 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:49:22:652.609 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:22:652.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:22:652.692 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691362652637 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:22:652.698 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:22:652.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:22:652.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691362652643 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:22:652.731 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:25:390.114 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:390.195 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691365389972 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 27 dequeues = 27 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 466 dequeues = 466 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 27 dequeues = 27 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 28 dequeues = 28 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 551 dequeues = 551 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691337433753 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691362652718 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:49:25:390.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:25:390.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:390.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691365390084 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 104 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 110 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 228 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 57 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 466, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:25:390.339 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:25:399.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:399.551 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691365399453 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.105 five_min_avg = 0.135 fifteen_min_avg = 0.135 ] cpu_share = 0.641667 ] ] 2025-09-01 Mon 01:49:25:399.571 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:49:25:399.579 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:399.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691365399465 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.641667 used_sys_mem = 2523939144 one_min_cpuload = 0.105 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:49:25:399.598 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:49:25:399.606 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:399.690 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691365399492 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6112 in_bytes = 1093908 out_tpkts = 5639 out_bytes = 1131224 raw_vhost_stats= [ name = vhost0 in_pkts = 6112 in_bytes = 1093908 out_pkts = 5639 out_bytes = 1131224 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5447 in_bytes = 1123160 out_pkts = 6115 out_bytes = 1093831 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:49:25:399.714 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:49:25:399.722 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:25:399.749 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691365399514 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:49:25:399.755 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:49:27:433.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:49:27:434.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:49:27:434.229 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:49:27:434.254 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691367433941 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:49:27:434.262 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:49:27:434.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:49:27:434.366 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:49:27:434.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:49:27:434.446 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691367434207 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 106 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 110 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 228 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 57 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 472, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:27:434.465 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:27:434.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:49:27:434.479 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:49:27:434.515 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:27:434.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Idle 2025-09-01 Mon 01:49:27:434.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:27:434.601 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691367434498 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:49:27:434.606 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:49:27:434.612 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:49:27:434.641 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:27:434.771 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691367434578 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 62 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 107 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 228 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 57 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 474, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:27:434.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:35:677.985 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:35:678.062 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691375677953 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:35:678.073 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:35:678.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:35:678.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691375678434 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 46661 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:35:678.504 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:35:678.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:49:35:678.742 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:35:678.786 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:35:678.805 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691375678777 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:35:678.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:35:678.816 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:35:678.823 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691375678783 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:35:678.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:51:621.232 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:51264 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:49:51:625.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:51270 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:49:53:972.793 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:53:972.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691393972759 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:53:972.988 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:53:972.996 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:53:973.015 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691393972915 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44953 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:53:973.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:53:973.091 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:49:53:973.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:53:973.183 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:53:973.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691393973176 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:53:973.209 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:53:973.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:53:973.222 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691393973181 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:49:53:973.228 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:49:55:390.347 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:55:391.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691395390213 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 28 dequeues = 28 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 484 dequeues = 484 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 28 dequeues = 28 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 29 dequeues = 29 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 572 dequeues = 572 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691367434543 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691393973214 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:49:55:391.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:55:391.866 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:55:391.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691395390312 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 108 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 115 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 236 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 484, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:55:392.015 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:55:399.681 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:55:399.804 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691395399652 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6274 in_bytes = 1121101 out_tpkts = 5797 out_bytes = 1157377 exception_packets = 38 exception_packets_allowed = 23 raw_vhost_stats= [ name = vhost0 in_pkts = 6274 in_bytes = 1121101 out_pkts = 5797 out_bytes = 1157377 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.54534e-14 total_out_bandwidth_utilization = 3.07154e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6540, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5666, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5602 in_bytes = 1149187 out_pkts = 6277 out_bytes = 1121024 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:49:55:399.827 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:49:55:399.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:55:399.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691395399679 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:49:55:399.867 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:49:57:434.643 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:49:57:435.140 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:49:57:435.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:49:57:435.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691397434726 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:49:57:435.294 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:49:57:435.294 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:49:57:435.337 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:49:57:435.306 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:49:57:435.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691397435220 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 110 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 115 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 236 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 488, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:57:435.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:49:57:435.496 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:49:57:435.502 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:49:57:435.535 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:49:57:435.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:49:57:435.602 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:57:435.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691397435518 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:49:57:435.625 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:49:57:435.632 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:49:57:435.658 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:49:57:435.753 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691397435595 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 111 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 116 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 236 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 59 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 490, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:49:57:435.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:03:218.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:03:219.046 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691403218899 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:03:219.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:03:219.126 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:03:219.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691403219116 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43119 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:03:219.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:03:219.316 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:50:03:219.389 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:03:219.434 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:03:219.455 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691403219424 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:03:219.462 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:03:219.468 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:03:219.476 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691403219433 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:03:219.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:24:753.311 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:24:753.382 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691424753283 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:24:753.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:24:753.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:24:753.464 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691424753447 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41517 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:24:753.470 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:24:753.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:50:24:753.738 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:24:753.789 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:24:753.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691424753782 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:24:753.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:24:753.834 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:24:753.842 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691424753786 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:24:753.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:25:390.570 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:390.640 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691425390432 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 29 dequeues = 29 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 500 dequeues = 500 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 29 dequeues = 29 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 30 dequeues = 30 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 591 dequeues = 591 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691397435561 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691424753833 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:50:25:390.652 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:25:390.663 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:390.787 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691425390544 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 65 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 112 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 118 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 244 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 500, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:25:390.820 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:25:399.836 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:399.889 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691425399808 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.035 five_min_avg = 0.11 fifteen_min_avg = 0.125 ] cpu_share = 0.65 ] ] 2025-09-01 Mon 01:50:25:399.906 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:50:25:399.914 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:399.926 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691425399819 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.65 used_sys_mem = 2523939144 one_min_cpuload = 0.035 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:50:25:399.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:50:25:399.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:400.038 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691425399844 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6412 in_bytes = 1136639 out_tpkts = 5923 out_bytes = 1180513 raw_vhost_stats= [ name = vhost0 in_pkts = 6412 in_bytes = 1136639 out_pkts = 5923 out_bytes = 1180513 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5725 in_bytes = 1172197 out_pkts = 6415 out_bytes = 1136562 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:50:25:400.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:50:25:400.062 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:25:400.084 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691425399867 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:50:25:400.090 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:50:27:435.638 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:50:27:436.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:50:27:436.245 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:50:27:436.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:50:27:436.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:50:27:436.274 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691427435718 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:50:27:436.681 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:50:27:436.691 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:50:27:436.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691427436217 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 65 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 114 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 118 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 244 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 506, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:27:436.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:27:436.844 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:50:27:436.849 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:50:27:436.885 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:27:436.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:50:27:436.959 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:27:436.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691427436868 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:50:27:436.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:50:27:436.984 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:50:27:437.018 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:27:437.141 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691427436947 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 66 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 115 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 119 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 244 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 508, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:27:437.160 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:33:729.611 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:33:729.664 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691433729570 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:33:729.673 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:33:729.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:33:729.777 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691433729756 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41167 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:33:729.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:33:729.988 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:50:33:730.049 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:33:730.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:33:730.143 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691433730093 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:33:730.149 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:33:730.155 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:33:730.162 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691433730096 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:33:730.178 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:51:645.096 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:39448 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:50:51:649.176 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:39452 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:50:54:454.147 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:54:454.206 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691454453918 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:54:454.217 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:54:454.224 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:54:454.234 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691454454118 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 41029 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:54:454.238 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:54:454.367 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:50:54:457.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:54:457.137 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:54:457.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691454457077 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:54:457.163 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:54:457.170 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:54:457.180 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691454457086 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:50:54:457.186 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:50:55:390.806 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:55:390.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691455390664 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 30 dequeues = 30 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 518 dequeues = 518 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 30 dequeues = 30 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 31 dequeues = 31 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 612 dequeues = 612 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691427436912 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691454457169 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:50:55:390.892 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:55:390.899 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:55:391.038 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691455390756 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 67 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 116 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 123 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 252 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 63 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 189, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 518, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:55:391.064 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:55:400.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:55:400.208 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691455400028 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6587 in_bytes = 1166590 out_tpkts = 6087 out_bytes = 1208513 exception_packets = 39 exception_packets_allowed = 24 raw_vhost_stats= [ name = vhost0 in_pkts = 6587 in_bytes = 1166590 out_pkts = 6087 out_bytes = 1208513 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.67761e-14 total_out_bandwidth_utilization = 3.28784e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6784, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 6065, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 5886 in_bytes = 1200071 out_pkts = 6590 out_bytes = 1166513 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:50:55:400.235 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:50:55:400.242 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:55:400.263 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691455400082 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:50:55:400.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:50:57:436.994 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:50:57:437.234 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:50:57:437.348 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:50:57:437.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:50:57:437.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:50:57:437.403 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691457437079 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:50:57:437.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:50:57:437.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:50:57:437.524 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691457437315 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 67 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 118 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 123 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 252 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 63 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 189, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 522, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:57:437.550 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:50:57:437.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:50:57:437.564 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:50:57:437.609 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:50:57:437.655 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:50:57:437.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:57:437.714 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691457437580 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:50:57:437.720 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:50:57:437.725 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:50:57:437.747 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:50:57:437.841 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691457437690 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 68 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 119 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 124 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 252 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 63 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 190, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 524, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:50:57:437.854 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:02:350.266 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:02:350.336 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691462350228 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:02:350.353 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:02:350.409 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:02:350.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691462350401 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34513 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:02:350.432 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:02:350.684 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:51:02:350.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:02:350.803 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:02:350.826 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691462350793 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:02:350.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:02:350.840 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:02:350.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691462350799 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:02:350.852 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:23:077.244 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:23:077.303 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691483077209 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:23:077.313 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:23:077.671 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:23:077.690 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691483077664 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36397 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:23:077.696 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:23:077.929 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:51:23:078.018 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:23:078.074 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:23:078.101 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691483078061 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:23:078.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:23:078.114 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:23:078.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691483078067 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:23:078.127 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:25:390.964 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:391.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691485390841 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 31 dequeues = 31 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 534 dequeues = 534 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 31 dequeues = 31 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 32 dequeues = 32 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 631 dequeues = 631 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691457437654 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691483078114 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:51:25:391.057 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:25:391.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:391.168 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691485390942 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 69 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 126 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 260 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 194, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 534, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:25:391.196 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:25:400.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:400.329 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691485400229 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.01 five_min_avg = 0.09 fifteen_min_avg = 0.115 ] cpu_share = 0.641667 ] ] 2025-09-01 Mon 01:51:25:400.341 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:51:25:400.351 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:400.370 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691485400251 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.641667 used_sys_mem = 2523939144 one_min_cpuload = 0.01 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:51:25:400.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:51:25:400.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:400.482 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691485400276 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6720 in_bytes = 1180710 out_tpkts = 6206 out_bytes = 1230452 raw_vhost_stats= [ name = vhost0 in_pkts = 6720 in_bytes = 1180710 out_pkts = 6206 out_bytes = 1230452 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 6002 in_bytes = 1221884 out_pkts = 6723 out_bytes = 1180633 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:51:25:400.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:51:25:400.506 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:25:400.526 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691485400299 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:51:25:400.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:51:27:437.741 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:51:27:437.944 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:51:27:438.073 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:51:27:438.099 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691487437812 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:51:27:438.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:51:27:438.117 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:51:27:438.147 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:51:27:438.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:51:27:439.090 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691487438045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 69 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 126 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 260 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 194, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 540, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:27:439.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:27:439.119 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:51:27:439.125 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:51:27:439.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:27:439.186 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:51:27:439.227 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:27:439.242 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691487439143 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:51:27:439.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:51:27:439.253 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:51:27:439.279 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:27:439.370 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691487439220 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 70 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 123 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 127 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 260 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 195, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 542, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:27:439.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:28:258.318 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.22:47958 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:51:32:590.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:32:591.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691492590923 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:32:591.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:32:591.448 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:32:591.494 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691492591423 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34047 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:32:591.505 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:32:591.661 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:51:32:591.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:32:591.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:32:591.846 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691492591806 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:32:591.855 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:32:591.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:32:591.892 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691492591814 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:32:591.898 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:51:668.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:38528 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:51:51:672.450 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:38534 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:51:54:938.203 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:54:938.271 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691514938173 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:54:938.283 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:54:938.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:54:938.623 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691514938600 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36649 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:54:938.630 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:54:938.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:51:54:938.878 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:54:938.916 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:54:938.938 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691514938906 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:54:938.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:54:938.965 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:54:938.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691514938911 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:51:54:938.978 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:51:55:391.254 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:55:391.340 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691515391076 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 32 dequeues = 32 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 552 dequeues = 552 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 32 dequeues = 32 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 652 dequeues = 652 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691487439186 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691514938964 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:51:55:391.359 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:55:391.370 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:55:391.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691515391208 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 71 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 132 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 268 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 202, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 552, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:55:391.511 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:55:400.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:55:400.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691515400441 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 6958 in_bytes = 1233170 out_tpkts = 6422 out_bytes = 1268669 exception_packets = 41 exception_packets_allowed = 26 raw_vhost_stats= [ name = vhost0 in_pkts = 6958 in_bytes = 1233170 out_pkts = 6422 out_bytes = 1268669 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 4.32976e-14 total_out_bandwidth_utilization = 4.81494e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 7987, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 8882, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 6215 in_bytes = 1259975 out_pkts = 6962 out_bytes = 1233135 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:51:55:400.638 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:51:55:400.646 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:55:400.669 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691515400488 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:51:55:400.679 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:51:57:439.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:51:57:439.471 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:51:57:439.597 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:51:57:439.610 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:51:57:439.649 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:51:57:439.623 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691517439337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:51:57:439.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:51:57:439.678 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:51:57:439.788 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691517439573 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 71 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 126 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 132 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 268 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 202, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 556, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:57:439.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:51:57:439.836 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:51:57:439.842 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:51:57:439.876 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:51:57:439.903 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:51:57:439.947 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:57:439.961 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691517439859 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:51:57:439.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:51:57:439.972 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:51:57:440.013 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:51:57:440.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691517439939 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 72 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 127 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 133 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 268 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 203, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 558, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:51:57:440.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:03:101.982 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:03:102.068 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691523101943 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:03:102.078 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:03:102.501 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:03:102.564 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691523102470 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40411 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:03:102.575 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:03:102.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:52:03:102.827 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:03:102.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:03:102.903 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691523102858 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:03:102.911 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:03:102.921 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:03:102.932 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691523102864 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:03:102.939 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:25:391.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:391.558 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691545391327 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 564 dequeues = 564 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 34 dequeues = 34 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 667 dequeues = 667 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691517439902 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691523102919 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:52:25:391.580 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:25:391.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:391.691 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691545391439 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 73 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 128 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 134 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 272 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 205, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 564, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:25:391.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:25:400.653 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:400.727 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691545400622 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.075 five_min_avg = 0.09 fifteen_min_avg = 0.115 ] cpu_share = 0.941667 ] ] 2025-09-01 Mon 01:52:25:400.735 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:52:25:400.743 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:400.755 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691545400637 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.941667 used_sys_mem = 2523939144 one_min_cpuload = 0.075 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:52:25:400.762 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:52:25:400.774 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:400.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691545400662 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 7107 in_bytes = 1250035 out_tpkts = 6554 out_bytes = 1293762 raw_vhost_stats= [ name = vhost0 in_pkts = 7107 in_bytes = 1250035 out_pkts = 6554 out_bytes = 1293762 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 6344 in_bytes = 1284942 out_pkts = 7111 out_bytes = 1250000 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:52:25:400.881 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:52:25:400.888 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:400.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691545400684 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:52:25:400.915 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:52:25:989.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:989.166 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691545989048 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:25:989.177 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:25:989.488 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:989.508 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691545989480 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34469 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:25:989.515 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:25:989.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:52:25:989.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:25:990.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:990.071 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691545989971 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:25:990.078 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:25:990.084 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:25:990.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691545989983 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:25:990.098 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:27:439.986 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:52:27:440.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:52:27:440.620 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:52:27:440.666 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:52:27:440.688 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:52:27:440.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691547440072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:52:27:440.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:52:27:440.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:52:27:440.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691547440599 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 73 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 130 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 135 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 276 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 69 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 207, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 574, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:27:440.850 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:27:440.859 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:52:27:440.864 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:52:27:440.897 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:27:440.920 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 01:52:27:440.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:27:440.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691547440882 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:52:27:440.985 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:52:27:440.991 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:52:27:441.026 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:27:441.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691547440955 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 74 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 131 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 136 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 276 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 69 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 208, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 576, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:27:441.130 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:33:343.033 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:33:343.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691553342997 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:33:343.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:33:343.183 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:33:343.194 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691553343178 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 46373 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:33:343.198 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:33:343.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:52:33:343.506 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:33:343.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:33:343.580 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691553343551 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:33:343.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:33:343.607 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:33:343.614 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691553343561 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:33:343.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:51:730.461 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:45240 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:52:51:735.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:45244 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:52:53:530.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:53:530.223 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691573530128 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:53:530.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:53:530.651 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:53:530.703 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691573530618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35349 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:53:530.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:53:530.876 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:52:53:530.976 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:53:531.043 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:53:531.061 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691573531033 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:53:531.085 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:53:531.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:53:531.103 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691573531039 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:52:53:531.109 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:52:55:391.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:55:391.752 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691575391552 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 34 dequeues = 34 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 586 dequeues = 586 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 34 dequeues = 34 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 35 dequeues = 35 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 692 dequeues = 692 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691547440920 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691573531091 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:52:55:391.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:55:391.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:55:391.873 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691575391658 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 75 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 132 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 140 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 284 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 214, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 586, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:55:391.898 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:55:400.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:55:401.008 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691575400827 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 7274 in_bytes = 1277174 out_tpkts = 6713 out_bytes = 1320337 exception_packets = 43 exception_packets_allowed = 28 raw_vhost_stats= [ name = vhost0 in_pkts = 7274 in_bytes = 1277174 out_pkts = 6713 out_bytes = 1320337 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 10729 out_bandwidth_usage = 10949 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.7161e-14 total_out_bandwidth_utilization = 3.18322e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 6855, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5872, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 6500 in_bytes = 1311391 out_pkts = 7279 out_bytes = 1277181 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:52:55:401.036 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:52:55:401.047 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:55:401.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691575400855 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:52:55:401.072 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:52:57:441.019 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:52:57:441.293 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:52:57:441.447 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:52:57:441.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:52:57:441.483 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:52:57:441.511 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691577441122 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:52:57:441.522 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:52:57:441.531 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:52:57:441.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:52:57:441.571 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:52:57:441.614 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Idle 2025-09-01 Mon 01:52:57:441.668 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:57:441.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691577441471 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 75 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 134 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 140 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 284 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 214, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 590, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:57:441.817 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:52:57:441.828 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:57:441.847 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691577441555 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:52:57:441.854 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:52:57:441.862 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:52:57:441.896 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:52:57:442.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691577441660 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 76 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 134 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 141 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 284 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 215, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 591, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:52:57:442.016 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:02:233.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:02:233.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691582233698 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:02:233.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:02:234.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:02:234.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691582234332 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38319 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:02:234.428 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:02:234.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:53:02:234.626 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:53:02:234.667 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:02:234.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691582234659 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:02:234.698 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:02:234.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:02:234.715 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691582234665 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:02:234.721 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:22:691.262 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:22:691.316 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756691602691154 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:22:691.333 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:22:692.083 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:22:692.111 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691602692071 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 34979 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:22:692.130 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:22:692.251 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:53:22:692.283 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:53:22:692.376 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:22:692.388 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691602692305 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:22:692.394 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:22:692.399 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:22:692.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691602692309 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:22:692.411 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:25:391.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:391.979 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691605391775 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 35 dequeues = 35 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 602 dequeues = 602 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 35 dequeues = 35 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 36 dequeues = 36 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 711 dequeues = 711 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691577441613 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691602692398 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:53:25:391.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:25:392.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:392.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691605391869 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 77 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 136 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 143 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 292 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 219, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 602, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:53:25:392.134 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:25:401.034 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:401.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691605401000 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.065 five_min_avg = 0.08 fifteen_min_avg = 0.105 ] cpu_share = 0.658333 ] ] 2025-09-01 Mon 01:53:25:401.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:53:25:401.123 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:401.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691605401013 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.658333 used_sys_mem = 2523939144 one_min_cpuload = 0.065 mem_res = 0 ], ] ] ] 2025-09-01 Mon 01:53:25:401.143 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 01:53:25:401.151 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:401.233 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691605401039 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 7833 in_bytes = 1412423 out_tpkts = 7203 out_bytes = 1390956 raw_vhost_stats= [ name = vhost0 in_pkts = 7833 in_bytes = 1412423 out_pkts = 7203 out_bytes = 1390956 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 6987 in_bytes = 1381884 out_pkts = 7838 out_bytes = 1412430 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:53:25:401.255 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:53:25:401.261 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:25:401.281 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691605401061 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:53:25:401.286 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:53:27:441.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:53:27:441.970 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:53:27:442.090 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:53:27:442.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691607441814 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:53:27:442.141 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:53:27:442.148 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:53:27:442.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:53:27:442.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:53:27:442.255 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691607442072 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 77 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 138 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 143 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 292 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 219, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 608, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:53:27:442.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:27:442.280 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 01:53:27:442.287 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 01:53:27:442.595 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:53:27:442.624 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Idle 2025-09-01 Mon 01:53:27:442.670 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:27:442.686 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691607442311 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:53:27:442.696 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:53:27:442.701 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 01:53:27:442.729 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:27:442.824 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691607442661 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 78 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 139 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 144 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 292 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 220, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 610, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:53:27:442.838 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:30:584.833 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:30:584.918 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691610584788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:30:584.930 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:30:585.369 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:30:585.406 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756691610585344 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 43005 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:30:585.414 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:30:585.747 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:53:30:585.832 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:53:30:585.889 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:30:585.917 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691610585870 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:30:585.925 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:30:585.932 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:30:585.951 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691610585876 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:30:585.958 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:51:312.463 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:51:312.544 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756691631312426 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:51:312.556 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:51:312.830 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:51:312.849 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691631312820 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45065 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:51:312.866 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:51:313.064 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 01:53:51:313.360 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 01:53:51:313.401 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:51:313.417 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691631313391 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:51:313.423 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:51:313.445 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:51:313.454 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756691631313398 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 01:53:51:313.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 01:53:51:757.888 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48792 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:53:51:762.991 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:48806 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 01:53:55:392.152 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:55:392.245 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691635392003 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 36 dequeues = 36 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 620 dequeues = 620 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 36 dequeues = 36 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 37 dequeues = 37 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 732 dequeues = 732 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756691607442624 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756691631313444 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 01:53:55:392.257 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:55:392.265 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:55:392.378 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756691635392122 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 140 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 148 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 300 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 226, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 620, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 01:53:55:392.412 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 01:53:55:401.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:55:401.490 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691635401245 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 8037 in_bytes = 1451560 out_tpkts = 7386 out_bytes = 1431142 raw_vhost_stats= [ name = vhost0 in_pkts = 8037 in_bytes = 1451560 out_pkts = 7386 out_bytes = 1431142 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 7.99057e-14 total_out_bandwidth_utilization = 1.26044e-13 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 14740, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 23251, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 7167 in_bytes = 1421944 out_pkts = 8042 out_bytes = 1451567 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 01:53:55:401.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 01:53:55:401.541 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 01:53:55:401.573 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756691635401302 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 01:53:55:401.581 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 01:53:57:442.689 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 01:53:57:442.918 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 01:53:57:443.026 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:53:57:443.055 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 01:53:57:443.069 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 01:53:57:443.095 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691637442766 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 01:53:57:443.102 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 01:53:57:443.110 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 01:53:57:443.215 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756691637443009 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 142 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 148 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 300 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 226, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 624, _iter59->first = sent _iter59->second = 0, ] ] ]