[07:59:43] YAY [08:01:59] * addshore starts looking at evil sql [08:16:38] That transaction enabled me to track down the edit that caused the deletion and also confirm my hypothosis :) https://phabricator.wikimedia.org/T237984#5750348 now i just have to keep staring at code [11:20:55] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [12:49:26] 10DBA, 10GlobalUsage, 10StructuredDataOnCommons: Normalize globalimagelinks table - https://phabricator.wikimedia.org/T241053 (10Ladsgroup) [13:03:16] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10jbond) [13:03:23] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) [13:03:48] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) [16:40:12] 10DBA, 10Wikimedia-Rdbms, 10Core Platform Team Workboards (Clinic Duty Team): Does an eqiad Mediawiki need to have codfw DB servers in its hostsByName? and vice versa - https://phabricator.wikimedia.org/T240991 (10CDanis) BTW I'm curious about both 1) the current need (or lack thereof) for cross-DC Mediawiki... [16:41:38] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team, 10Patch-For-Review: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) @Anomie if you can review https://gerrit.wikimedia.org/r/558851 that's what's left to be able to close this task \o/ [16:42:25] 10DBA, 10Wikimedia-Rdbms, 10Core Platform Team Workboards (Clinic Duty Team): Does an eqiad Mediawiki need to have codfw DB servers in its hostsByName? and vice versa - https://phabricator.wikimedia.org/T240991 (10jcrespo) As I mentioned T134809 is probably the most relevant ticket (although not the only one). [16:42:28] 10DBA, 10Wikimedia-Rdbms, 10Core Platform Team Workboards (Clinic Duty Team): Does an eqiad Mediawiki need to have codfw DB servers in its hostsByName? and vice versa - https://phabricator.wikimedia.org/T240991 (10Anomie) I don't see any code using entries in hostsByName that aren't listed in one of sectionL... [18:22:13] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [18:22:57] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [18:36:35] 10DBA, 10Wikimedia-Rdbms, 10Core Platform Team Workboards (Clinic Duty Team): Sync understanding of MediaWiki rdbms 'weight' behaviour with DBAs - https://phabricator.wikimedia.org/T239900 (10CDanis) [18:36:37] 10DBA, 10Wikimedia-Rdbms, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Does an eqiad Mediawiki need to have codfw DB servers in its hostsByName? and vice versa - https://phabricator.wikimedia.org/T240991 (10CDanis) 05Open→03Resolved a:03CDanis [22:12:29] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Jclark-ctr) Confirmed: Service Request 1007375142 was successfully submitted