[08:47:52] hello hello [08:48:10] a lot of purged daemons on cp* experienced an hour of backlog like [08:48:11] https://grafana.wikimedia.org/d/RvscY1CZk/purged?orgId=1&var-datasource=esams%20prometheus%2Fops&var-cluster=cache_text&var-instance=cp3058&from=1606619696493&to=1606627736195 [08:48:23] from ~3:50/4:00 UTC to ~5 UTC [08:49:09] I don't see relevant logs from the purged side, and the eqiad kafka topic shows an increase (not dramatic though compared to the otherdays): [08:49:16] https://grafana.wikimedia.org/d/000000234/kafka-by-topic?orgId=1&from=1606616585786&to=1606626801635&var-datasource=eqiad%20prometheus%2Fops&var-kafka_cluster=main-eqiad&var-kafka_broker=All&var-topic=eqiad.resource-purge [08:51:40] possibly ATS being slow in processing extra purges? [09:28:32] I found a lot of events related to ru.m.wikinews.org's purges, might have been it [09:32:12] -- [09:32:36] side note - kafkacat has a new nice option to start from a timestamp, like [09:33:01] kafkacat -C -b kafka-jumbo1001.eqiad.wmnet -t eqiad.resource-purge -o s@1601425805519 [09:34:01] but not in our version (1.3).. on Bullseye we have 1.6.0, I am wondering if the package maintainer could add a backport for buster [09:34:48] paravoid: --^ :D [09:34:54] opening a task [09:39:47] ttps://phabricator.wikimedia.org/T268936