[04:25:01] DBA, MediaWiki-Database, Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#2451214 (RobLa-WMF) [13:01:39] DBA, Expiring-Watchlist-Items, MediaWiki-Watchlist, TCB-Team, and 3 others: Add wl_timestamp to the watchlist table - https://phabricator.wikimedia.org/T125991#2452494 (Addshore) [14:47:13] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453162 (jcrespo) [14:49:49] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453190 (jcrespo) p:Triage>Unbreak! I've setup a watchdog that kills API queries running for over 10 seconds. Other queries that are misbehaving: ``` SELECT /* ApiQueryRecen... [14:53:18] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453211 (jcrespo) https://logstash.wikimedia.org/#dashboard/temp/AVXflyIUw3dCNxx2LVk8 [14:58:19] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453217 (jcrespo) It started around 36 hours ago, but got worse in the last 12 hours: https://tendril.wikimedia.org/report/slow_queries_checksum?checksum=4d1857ea652abc7849668dc00f80... [15:16:19] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453333 (jcrespo) Most queries seem to come from a few zh-users: - 写字板 - Stang - A2093064 - Assanges - Galaxyharrylion - Amazingloong (there may be some false positives... [15:32:23] DBA, MediaWiki-API: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453162 (Anomie) In general, this reminds me of T101502 and similar bugs: the module has a bunch of filters, and if rows passing the combination of filters is relatively rare it can r... [15:40:30] DBA, MediaWiki-API, Patch-For-Review: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2453478 (jcrespo) @Anomie yes, however, knowing that happens I think we should be prepared- have some kind of limitation on the query time when the same query is... [17:03:39] DBA, MediaWiki-API, Patch-For-Review: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2454158 (jcrespo) The explain is very clean: ``` MariaDB PRODUCTION s2 localhost zhwiki > EXPLAIN SELECT /* ApiQueryRecentChanges::run */ rc_id, rc_timest... [17:26:08] DBA, Analytics: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2454271 (mforns) @jcrespo I finished updating the white-list. Sorry for the delay, I needed to make sure that changes to schemas between 2015's audit and today were included and that the respective schem... [17:27:23] DBA, Analytics: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2454286 (mforns) a:jcrespo @jcrespo Please, let me know if the white-list is what you expected and what can I do to help you in the next steps. Thanks! [17:29:04] DBA, MediaWiki-API, Patch-For-Review: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2454290 (jcrespo) I tested extensively the queries, and I agree with your conclusions, but I have some extra: * I have fixed a bug that I think it wasn't part o... [17:41:00] DBA, MediaWiki-API, Patch-For-Review: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2454316 (jcrespo) p:Unbreak!>High High because I think it will happen again, but I am now prepared. [17:44:31] DBA, Analytics: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2454323 (jcrespo) Please create a change review on puppet/operations (anywhere, I will move it somewhere else) instead of attaching it here, I will take it from here. [18:10:20] DBA, Patch-For-Review: Reimage m3 nodes db1043 and db1048 and upgrade them to mariadb10 - https://phabricator.wikimedia.org/T126352#2454467 (jcrespo) [18:10:23] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2454469 (jcrespo) [18:10:47] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2401019 (jcrespo) Remember to revert https://gerrit.wikimedia.org/r/269447 [18:23:14] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2454534 (jcrespo) I think I have fixed all slave differences between m3-master and m3-slave. Most were false positives due to 5.5 and 10 or tool limitations, but there w... [22:59:33] DBA, Operations, hardware-requests, ops-codfw, Patch-For-Review: Decommission es2005-es2010 - https://phabricator.wikimedia.org/T134755#2455905 (RobH) Open>Resolved switch port descriptions removed