mirror of
https://github.com/outbackdingo/patroni.git
synced 2026-08-26 15:40:21 +00:00
Alexander KukushkinandGitHub
66d77697ae
There is an opinion that LIST requests with labelSelector to K8s API are expensive and Patroni was doing two such requests per HA loop (LIST pods and LIST endpoints/configmaps). To efficiently detect object changes we will switch to the LIST+WATCH approach. The initial LIST request populates the ObjectCache and events from the WATCH request update it. In addition to that, the ObjectCache will be updated after performing the UPDATE operations on the K8s objects. To avoid race conditions, all operations on ObjectCache are performed after comparing the resource_version of the old and the new objects and rejected if the new resource_version value is smaller than the old one. The disadvantage of such an approach is that it will require keeping three connections to the K8s API from each Patroni Pod (previously it was two). Yesterday I deployed this feature branch on our biggest K8s cluster, with ~300 Patroni pods. The CPU Utilization on K8s master nodes immediately dropped from ~20% to ~10% (two times), and the incoming traffic on master nodes dropped ~7-8 times! Last, but not least, we get more or less the same impact on etcd cluster behind K8s master nodes, the CPU Utilization dropped nearly twice and outgoing traffic ~7-8 times.