[09:11:05] hello hello [09:11:28] around midnight UTC there has been a big mcrouter TKO event [09:11:29] https://grafana.wikimedia.org/d/000000549/mcrouter?orgId=1&from=1607208820145&to=1607215010799&var-source=eqiad%20prometheus%2Fops&var-cluster=appserver&var-instance=All&var-memcached_server=All [09:11:35] seems for appservers [09:12:19] and mc1021 showed a big tx saturation jump at the same time [09:12:19] https://grafana.wikimedia.org/d/000000316/memcache?viewPanel=56&orgId=1&from=1607208130325&to=1607215000050 [09:14:13] that was slab-147 related [09:14:14] https://grafana.wikimedia.org/d/000000317/memcache-slabs?viewPanel=60&orgId=1&from=1607207963806&to=1607215892920&var-datasource=eqiad%20prometheus%2Fops&var-cluster=memcached&var-instance=mc1021&var-slab=All [09:15:31] around 140k in size, probably the usual sqlblobstore [09:17:52] gutter took over, all good [09:25:08] also very interestingly, mc1022's tx bw usage is still constantly ~+100Mbps than the rest of the shards, started at the beginning of October with https://en.wikipedia.org/w/index.php?oldid=983013703 [09:25:42] (the mw appservers are hammering mc1022 for a related sqlblobstore key)