[07:08:02] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Degraded RAID on db1103 - https://phabricator.wikimedia.org/T275266 (10wiki_willy) Ack @Marostegui, we'll take a look at it, with whoever heads onsite first this week.  @Cmjohnson or @Jclark-ctr - since this machine is out of warranty, can you see if you can grab a spare drive from...
[07:08:11] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Degraded RAID on db1103 - https://phabricator.wikimedia.org/T275266 (10wiki_willy) a:03Jclark-ctr
[15:55:59] <wikibugs>	 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10RhinosF1)
[15:58:04] <wikibugs>	 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10RhinosF1) >>! In T258361#6830518, @Marostegui wrote: > db1162 is fully pooled  This just went down.
[16:07:12] <marostegui>	 RhinosF1: I am creating a task for that ^
[16:07:15] <marostegui>	 Just depooled the host
[16:07:22] <marostegui>	 The idrac is unavailable so I cannot even check the HW logs
[16:07:53] <wikibugs>	 10DBA, 10ops-eqiad: db1162 crashed - https://phabricator.wikimedia.org/T275309 (10Marostegui)
[16:08:14] <wikibugs>	 10DBA, 10ops-eqiad: db1162 crashed - https://phabricator.wikimedia.org/T275309 (10Marostegui)
[16:08:16] <wikibugs>	 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui)
[16:08:22] <wikibugs>	 10DBA, 10ops-eqiad: db1162 crashed - https://phabricator.wikimedia.org/T275309 (10Marostegui) p:05Triage→03Medium
[16:09:15] <marostegui>	 sobanski: ^
[16:10:20] <marostegui>	 Having the mgmt iface also unavailable is strange...and probably not a good sign 
[16:10:30] <marostegui>	 Anyways, nothing we can do for now, we need onsite help
[16:15:37] <wikibugs>	 10DBA, 10decommission-hardware: decommission db1076.eqiad.wmnet - https://phabricator.wikimedia.org/T274752 (10Marostegui) 05Open→03Stalled The replacement for this host, db1162, crashed so let's not depool this one for now: T275309
[16:15:40] <wikibugs>	 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui)
[16:16:54] <RhinosF1>	 marostegui: ack, probably best given Sunday
[16:17:48] <RhinosF1>	 Enjoy what's left of weekend
[16:17:55] <marostegui>	 same to you!!! 
[16:17:56] <marostegui>	 thanks
[16:19:19] <RhinosF1>	 Np