[12:59:54] DBA, Analytics, Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2457599 (mforns) @jcrespo Please, see above the patch I created. When the file has a permanent location in puppet, I will add some documentation on EventLogging's Wikitech page. Th... [13:41:09] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2457774 (mmodell) @jcrespo I think the phabricator_search tables are non-critical as they can be re-generated from other data. I think that at least some of the migrati... [13:49:24] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2457801 (chasemp) The idea was for `rt_migration` and `bugzilla_migration` to live "forever" or at least as long as anyone cared that at one time we used either. Those... [14:06:57] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2457840 (jcrespo) > The idea was for rt_migration and bugzilla_migration to live "forever" or at least as long as anyone cared that at one time we used either. That is... [16:40:05] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2458516 (jcrespo) I got `fab_migration` and `rt_migration` mixed. No need to ask anything else- I will archive and drop fab_migration, percona and test. I will be waiti... [16:40:20] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2458517 (chasemp) no worries, noted in irc I meant to keep rt_migration too :) This go hand in hand w/ https://phabricator.wikimedia.org/diffusion/OPUP/browse/productio... [18:06:16] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2458980 (mmodell) @jcrespo: There is a scheduled downtime every Thursday at 01:00 (AM) GMT. If you want to pick a better time for your convenience then let me know when... [18:12:40] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2459003 (jcrespo) Assuming everybody knows this is happening- This is master failover- we are going to take m3 and point it somewhere else. While connections finish (an... [18:13:32] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2459022 (jcrespo) Assuming everybody knows this is happening (disruptive maintenance happens every week)- we can do it tomorrow. [18:31:56] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2459164 (mmodell) @jcrespo: Yes there is a one hour scheduled maintenance which is planned downtime although usually it takes no more than a few minutes. I'm game when y... [19:37:32] DBA, Operations, ops-eqiad: db1034 lag - https://phabricator.wikimedia.org/T139280#2459362 (Cmjohnson) [20:35:08] DBA, MediaWiki-API, Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459623 (Halfak) [20:37:43] DBA, MediaWiki-API, Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459659 (Halfak) I just tried a similar query against enwiki. `https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=ids|user|timestamp|userid... [20:44:28] DBA, MediaWiki-API, Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459623 (Reedy) ``` ApiPageSet::initFromRevIDs 10.64.48.20 2013 Lost connection to MySQL server during query (10.64.48.20) SELECT rev_id,rev_page FROM `revi... [20:47:40] DBA, MediaWiki-API, Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459710 (Reedy) ``` mysql:wikiadmin@db1080 [enwiki]> explain SELECT rev_id,rev_page FROM `revision`,`page` WHERE rev_id IN ('21914234230','3243242','234324... [23:43:36] DBA, MediaWiki-API, Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459623 (Anomie) I've seen a bug like this before... There it is: T131026#2187446. Apparently having an int-as-string that exceeds the size of the database fie... [23:48:17] DBA, Collaboration-Team-Interested, MediaWiki-extensions-CentralAuth, Notifications, Wikimedia-log-errors: CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks - https://phabricator.wikimedia.org/T121161#2460281 (Mattflaschen-WMF) [23:49:14] DBA, Collaboration-Team-Interested, MediaWiki-extensions-CentralAuth, Notifications, and 2 others: CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks - https://phabricator.wikimedia.org/T121161#1871031 (Mattflaschen-WMF) @jcrespo Any idea what may be causing this deadlock?