[06:30:59] https://phabricator.wikimedia.org/P9211 weird [06:31:21] I am about to reply to the main ticket [06:31:38] not sure what is the main ticket [06:31:47] the new one that was created yesterday I mean [06:33:51] this is something else [06:36:03] https://phabricator.wikimedia.org/T233986#5528640 [09:32:56] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) @Cmjohnson did this happen yesterday in the end? [09:35:21] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Marostegui) >>! In T233698#5527397, @Cmjohnson wrote: > @Marostegui Can you depool it leave it for us to do when we get a free moment. It's an easy thing to do... [10:43:12] 10DBA: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Marostegui) p:05Triage→03Normal [10:44:59] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Cmjohnson) @Marostegui the tech couldn’t make it in time yesterday and were scheduled today 1100-1300 local time. [10:45:32] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) >>! In T229452#5529113, @Cmjohnson wrote: > @Marostegui the tech couldn’t make it in time yesterday and were scheduled > today 1100-1300 local time. Great!... [10:48:51] 10DBA, 10Patch-For-Review: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [12:25:48] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Ladsgroup) I think the reason that it doesn't need partitioning anymore is {T184485} (and things like {T189596}) which dropped 1.4B rows from logging tables. ([[https://docs.google.com/spreadsheets/d/1EKI... [12:35:50] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team: Prepare and check storage layer for nqowiki - https://phabricator.wikimedia.org/T230543 (10Marostegui) a:05Marostegui→03None db1124 (sanitarium), db2094 (sanitarium), labsdb1009, 1010, 1011 and 1012 are clean. I have created the `nqowiki_p` o... [12:59:59] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [13:00:23] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [13:01:26] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) p:05Triage→03Normal [13:21:25] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [13:36:39] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [13:40:41] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) a:03Marostegui >>! In T233625#5529328, @Ladsgroup wrote: > I think the reason that it doesn't need partitioning anymore is {T184485} (and things like {T189596}) which dropped 1.4B rows from... [13:40:56] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) a:03Marostegui [14:02:18] 10DBA, 10Operations, 10ops-eqiad: db1074 crashed: Broken BBU - https://phabricator.wikimedia.org/T231638 (10Jclark-ctr) [14:03:30] 10DBA: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Andrew) a:05Andrew→03None [15:37:49] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Daimona) [15:42:27] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Daimona) [17:02:51] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Anomie) [20:14:42] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) I see this host is back up, so I guess the mainboard has been replaced?