[05:57:15] <wikibugs>	 10DBA, 10Operations: snapshot for s6/s7 at eqiad taken more than 4 days ago - https://phabricator.wikimedia.org/T234152 (10Marostegui) ` 05:15:01 <+icinga-wm> RECOVERY - snapshot of s6 in eqiad on db1115 is OK: snapshot for s6 at eqiad taken less than 4 days ago and larger than 90 GB: Last one 2019-09-30 04:17...
[05:58:29] <wikibugs>	 10DBA, 10Operations: snapshot for s6/s7 at eqiad taken more than 4 days ago - https://phabricator.wikimedia.org/T234152 (10Marostegui) p:05Triage→03Normal
[05:58:58] <wikibugs>	 10DBA, 10Operations: snapshot for s6/s7 at eqiad taken more than 4 days ago - https://phabricator.wikimedia.org/T234152 (10jcrespo) a:03jcrespo
[06:25:55] <wikibugs>	 10DBA, 10Operations: snapshot for s6/s7 at eqiad taken more than 4 days ago - https://phabricator.wikimedia.org/T234152 (10jcrespo) There is some amount of "self healing", but I will rerun manually some backups so not to miss the window. Thanks @jijiki for the report. These alerts are like RAID alerts- they ar...
[07:14:32] <wikibugs>	 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) p:05Triage→03Normal
[07:14:59] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) p:05Triage→03Normal
[07:16:02] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) 05Open→03Stalled Stalling until wmf.25 is out: https://tools.wmflabs.org/versions/
[09:09:13] <wikibugs>	 10DBA, 10Core Platform Team Legacy (Watching / External), 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.34-notes (1.34.0-wmf.24; 2019-09-24), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) >>! In T2...
[14:19:45] <wikibugs>	 10DBA, 10Core Platform Team, 10MW-1.34-notes (1.34.0-wmf.24; 2019-09-24), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Anomie)
[14:29:41] <wikibugs>	 10DBA, 10Analytics, 10CheckUser, 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Schema changes for `cu_changes` and `cu_log` table - https://phabricator.wikimedia.org/T233004 (10Milimetric)
[14:58:34] <wikibugs>	 10DBA, 10Core Platform Team, 10MW-1.34-notes (1.34.0-wmf.24; 2019-09-24), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Anomie) Yes, it's included in wmf/1.34.0-wmf.24. As of about 19:23 UTC on 2019-09-26, according to https...
[16:47:10] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Cmjohnson) @marostegui can you please depool this. I will do around 1400UTC on Thursday
[16:56:00] <wikibugs>	 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui)
[16:57:36] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Marostegui) >>! In T233698#5534931, @Cmjohnson wrote: > @marostegui can you please depool this. I will do around 1400UTC on Thursday  Will have it ready by Thurs...
[20:01:01] <wikibugs>	 10DBA, 10Operations: snapshot for s6/s7 at eqiad taken more than 4 days ago - https://phabricator.wikimedia.org/T234152 (10jcrespo) 05Open→03Resolved ` root@db1115.eqiad.wmnet[zarcillo]> SELECT * FROM backups WHERE section in ('s6', 's7') ORDER BY id desc LIMIT 5; +------+----------------------------------...