One Bite At A Time
Posted on 2023-12-21

As part of our regular Gradle build scan investigation we looked at an AndroidX build that should have been largely a 100% remote cache hit. We found that we were in fact hitting the cache as expected (yay!), but the build timeline had a long task that hit the cache and yet was taking way longer than any other task (nay!).

Screenshot of Gradle timeline showing one task taking a long time (timeline)

Taking a deeper look, we noticed that this task has a fairly large 60.7MB output, but it is downloading it at a measly 750KBps.

Screenshot of Gradle task detail view (task details)

Luckily it was easily reproducible locally. Trying to download the exact same cache entry using gsutil directly took <2 seconds, so we clearly had an issue. Taking a cursory look at a network activity on the machine showed that even Gradle build cache download takes little time, but the cache entry is not returned for over a minute after.

Screenshot of network manager showing one big burst

Then following profiling that I’ve blogged about before, it was clear that we were spending all of our time in

Screenshot of a flamegraph for fetching the cache entyr

And then it dawned on us, we were not doing any sort of buffering for this!!! A small PR later we went from 82s (750KBps) to 1.7s (36.8MBps) to download the same remote cache entry.

Moral of the story? Always be buffering your input streams! How did I not learn this from I/O profiling earlier this year?