[07:15:19] <wikibugs>	 10Traffic, 10Operations, 10ops-esams: Degraded RAID on lvs3001 - https://phabricator.wikimedia.org/T166965#3399619 (10Volans) I've commented out the `MAILADDR` line to avoid to get one email per day. Given that we have also the Icinga check we could consider to comment it out broadly across the fleet. The fi...
[07:33:16] <wikibugs>	 10Traffic, 10Operations, 10ops-esams: Degraded RAID on lvs3001 - https://phabricator.wikimedia.org/T166965#3399661 (10Volans) And of course that was not enough, I had to also add an `exit 0` to `/etc/cron.daily/mdadm` to prevent it from running, without the `MAILADDR` setting the report check refuses to run...
[08:51:47] <wikibugs>	 10Traffic, 10netops, 10Operations: codfw row B switch upgrade - https://phabricator.wikimedia.org/T169345#3399920 (10Marostegui) In this row, as important hosts we have to downtime:  es2018 -> es1014 needs to be downtimed as it will page with replication broken db2029 is s7 codfw master (We might want to dow...
[09:12:41] <wikibugs>	 10Traffic, 10netops, 10Operations: codfw row B switch upgrade - https://phabricator.wikimedia.org/T169345#3400023 (10elukey) For kafka2002 it is sufficient to depool it from eventbus via pybal/conftool, and then re-balance the cluster when the work is done (will take care of the two steps).
[09:26:33] <wikibugs>	 10Traffic, 10netops, 10Operations: codfw row B switch upgrade - https://phabricator.wikimedia.org/T169345#3400075 (10elukey) During the last upgrade we have generated a lot of pages and alerts on IRC, that is expected since the maintenance needed reboots of all the row's switches. What I am wondering is if w...
[10:44:40] <wikibugs>	 10Traffic, 10netops, 10Operations: codfw row B switch upgrade - https://phabricator.wikimedia.org/T169345#3400320 (10ayounsi) During the first upgrade, despite the switch being downtimed in Icinga, the upgrade process didn't make it go down (as in it was still replying to pings) so all the hosts depending on...
[17:35:08] <wikibugs>	 10Traffic, 10Operations, 10RESTBase, 10RESTBase-API, 10Services (next): RESTBase support for www.wikimedia.org missing - https://phabricator.wikimedia.org/T133178#3401854 (10GWicke) To @bblack's concern about scoping on projects vs. all *.wikimedia.org domains:   So far, both the portal at www.wikimedia....