[09:29:21] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-API, 05MW-1.28-release-notes, and 2 others: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2511172 (10jcrespo) This change has been deployed to all wikis/all servers. I am considering this resolved in term... [10:59:16] 10DBA, 10Dumps-Generation, 13Patch-For-Review: db1053 transient replica lag during dumping - https://phabricator.wikimedia.org/T132279#2511279 (10ArielGlenn) Today's dump skipped over that stage and hung on a dependency, no info in the logs, which is problematic. It skipped over the tables job as well. I'v... [11:01:13] 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2511287 (10ArielGlenn) Well at the risk of being... [11:54:30] 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2511355 (10jcrespo) Do not link to the testing, w... [13:02:29] 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2511457 (10ArielGlenn) Awesome, thanks! [13:23:47] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes, 15User-Josve05a: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511468 (10jayvdb) [13:39:49] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes, 15User-Josve05a: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511468 (10jcrespo) 10.64.16.102 is db1084. I do not see anything wrong with the server itself, although I... [13:44:29] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes, 15User-Josve05a: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511526 (10jcrespo) p:05Normal>03High Indeed, `AND (img_timestamp<'20160601')` makes the query go from... [14:06:15] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes, 15User-Josve05a: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511539 (10jcrespo) The reason being that results with rc_patrolled = '0' are very rare on old results, so... [14:11:30] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511544 (10Josve05a) [14:12:49] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511546 (10jayvdb) > A third option is to disallow hidepatrolled=1 + offset=* - you can hide patrolled edits on the latest cha... [14:13:03] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511547 (10matmarex) I'll note that there will not even be any results older than `$wgRCMaxAge` seconds, because `recentchange... [14:14:45] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511552 (10jcrespo) >>! In T141765#2511546, @jayvdb wrote: >> A third option is to disallow hidepatrolled=1 + offset=* - you c... [14:17:27] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511560 (10jcrespo) I want to hear @Anomie's take on this, he is aware of many issues regarding STRAIGHT_JOINs on mediawiki (w... [14:19:55] 10DBA, 06Labs, 10Labs-Infrastructure: Users can't run EXPLAIN queries to check the theoretical efficiency of their SQL - https://phabricator.wikimedia.org/T141095#2511565 (10chasemp) The idea w/ associating with T140788 is we want to approach this issue as part of the rearchitecting. Removing this as a subt... [14:20:25] 10DBA, 06Labs, 10Labs-Infrastructure, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2475959 (10chasemp) [14:20:27] 10DBA, 06Labs, 10Labs-Infrastructure: Users can't run EXPLAIN queries to check the theoretical efficiency of their SQL - https://phabricator.wikimedia.org/T141095#2511578 (10chasemp) [14:21:38] 10DBA, 06Labs, 10Labs-Infrastructure, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2475959 (10chasemp) [14:21:40] 10DBA, 06Labs, 10Labs-Infrastructure: Users can't run EXPLAIN queries to check the theoretical efficiency of their SQL - https://phabricator.wikimedia.org/T141095#2486960 (10chasemp) 05duplicate>03Open [14:27:08] 10DBA, 06Labs, 10Labs-Infrastructure: Explore 'Analyze' statement as substitute for Explain - https://phabricator.wikimedia.org/T141095#2511585 (10chasemp) [14:34:34] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511587 (10jayvdb) [14:35:51] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511468 (10jayvdb) >>! In T141765#2511547, @matmarex wrote: > I'll note that there will not even be any results older than `$w... [14:41:03] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2511591 (10jcrespo) @Nikerabbit Allow me to suggest talking to the people involved in T120451. We do not need to have a perfect plan that will never... [15:05:41] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) - https://phabricator.wikimedia.org/T141765#2511468 (10jcrespo) [16:17:30] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2511983 (10jcrespo) 05Resolved>03Open [16:18:13] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#1749178 (10jcrespo) Reopened and merging T141340 here. It is not urgent, but needs investigation. [16:18:57] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2511990 (10jcrespo) [16:19:00] 10DBA, 10MediaWiki-Database, 10MediaWiki-JobRunner, 10MediaWiki-extensions-Renameuser, 07Performance: /* RenameUserJob::run */ of user Burgundo created >1000 seconds of lag on itwiki (db1036), impacting recentchanges, watchlist and log visualization of al... - https://phabricator.wikimedia.org/T141340#2494829 [17:26:30] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2512365 (10greg) a:03dduvall Assigning to Dan per meeting discussion.