Interesting high latencies found when migrating search service to k8s

This is a followup to https://www.reddit.com/r/kubernetes/comments/1imbsx5/moving\_a\_memory\_heavy\_application\_to\_kubernetes/. I was asking if its expected that moving a search service to k8s would spike latencies. It was good to hear that its not expected that k8s would significantly reduce performance.

When we migrated, we found that every few min, there is a request that takes 300+ms or even a few seconds to complete, for a p9999 of 30ms.

After a lot of debugging, we disabled cadvisor and the high spike latencies resolved. Cadvisor runs with default settings and 30s intervals. We use it to monitor a lot of system stats.

This thread is to see if anyone has ideas? Given that ultimately root causing this is likely not worth it work wise, its just personal interest now to see if I can find the root cause. I'm wondering if anyone has any ideas on this.

Some data points:

- Our application itself uses fbthrift for server and thread management. the io threads use epoll_wait and the cpu threads use futex and spinlocks. The work themselves accesses a large mmap file for random reads that is mlocked into memory. Overall from an OS point of view, its not a very complicated application.

- The only root cause that I can think of is lock contention. Tuning the cfs_period_us for the cfs to a higher value (625ms vs 100ms default) also resolved the issue which points to some type of lock contention + pre-emption issue, where lock holders getting pre-empted also causes lock waiters to time out for the current time slice. But cadvisor and our application don't share any locks that i'm aware of.

- The search application does not make any sysfs calls.

- CPU pinning for isolation also did not result the issue, pointing to some type of kernel call issue.