We maintain a handy androidx.dev artifactory service that allows developers to try out
androidx libraries at HEAD
to validate fixes or just live on a bleeding edge. In fact, it is also used by
Android Studio team to serve Android Gradle Plugin at HEAD
too.
Last week, I looked at our traffic to this service, and I was kind of blown away that we were spiking to over 600
requests per second. The request pattern was cyclical with about 15-minute period. A deeper analysis showed that
majority of this traffic was coming from bots (e.g. meta-externalagent
, Barkrowler
, and others) going through our
endless build pages that we create every time a change lands and navigating to every .xml
, .pom
and other files.
All of was creating a heavy load on our service while providing no benefit as all of these builds are intended to
be short-lived.
We added a robots.txt
to tell bots to stop crawling our pages and within a day
our request traffic dropped drastically.
It seems that we are largely back to real users and some bots that don’t respect robots.txt
.