[06:32:15] I have finished disabling codfw -> eqiad replication. It all went well, maintenance tasks can resume [08:04:50] Emperor: ms-fe1010.eqiad.wmnet is struggling to accept connections [08:07:46] vgutierrez@lvs1020:~$ sudo -i journalctl -u pybal --since=today --grep ms-fe1010 |grep ERROR |wc -l [08:07:46] 137 [08:08:41] another PoV for the same issue: https://grafana.wikimedia.org/goto/SQuwk9hNR?orgId=1 [08:09:05] started yesterday at 14:11 UTC [08:29:33] thanks, I'll give it a kick [08:55:54] ugh, not awake yet. Actually kicked 1010 (not 2010) this time 🤦 [08:56:10] hmm the impacted one is 1010 [08:56:13] on eqiad [08:56:30] yes, exactly, I got there in the end [08:56:42] oh you hit 2010 first, gotcha [08:56:50] * vgutierrez sends some ☕ to Emperor [09:52:43] old binlogs of x2 is getting rotated out while it doesn't get that much write anymore https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=1h&var-server=db2142&var-datasource=thanos&var-cluster=mysql&from=now-30d&to=now&viewPanel=28 [09:53:11] it'll go down to much lower in three weeks