[05:36:40] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Patch-For-Review, 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) [05:36:56] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [05:36:58] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Patch-For-Review, 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) 05Open→03Resolved All done [05:37:00] 10DBA, 10Schema-change, 10Tracking-Neverending: [DO NOT USE] Schema changes for Wikimedia wikis (tracking) [superseded by #Blocked-on-schema-change] - https://phabricator.wikimedia.org/T51188 (10Marostegui) [05:38:15] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [06:01:09] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [06:02:30] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) I have removed the following grants that were for dbproxy1005, from m5 databases (and also checked they don't exist anywhere else): ` # host 10.64.16.155 155.16.64.10.in-addr.arpa dom... [06:03:13] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [06:03:35] 10DBA, 10Operations: Decommission dbproxy1005.eqiad.wmnet - https://phabricator.wikimedia.org/T231967 (10Marostegui) [06:15:36] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1017 is now ready to replace dbproxy1005 (even though they are not in use at the moment) ` # mysql --skip-ssl -hdbproxy1017 -e "select @@hostname" +------------+ | @@hostname |... [06:15:38] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [06:39:16] 10DBA, 10Operations, 10Patch-For-Review: Decommission dbproxy1005.eqiad.wmnet - https://phabricator.wikimedia.org/T231967 (10Marostegui) [07:24:11] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) I have been talking to Jaime about this, and we might better wait for https://gerrit.wikimedia.org/r/#/c/operations/puppe... [07:47:16] 10DBA, 10Operations, 10Patch-For-Review: Drop puppet database from m1 - https://phabricator.wikimedia.org/T231539 (10Marostegui) The following grants have been dropped: ` root@db1135.eqiad.wmnet[(none)]> drop user if exists 'puppet'@'10.64.0.165'; Query OK, 0 rows affected (0.00 sec) root@db1135.eqiad.wmnet... [08:07:09] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) a:05Marostegui→03None [09:20:21] I filed https://phabricator.wikimedia.org/T232081 [09:25:39] Ah, thanks, I saw it today in the morning and added it to my today's to-do [09:25:40] thanks [13:00:48] 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) This is kinda the same behaviour we have: https://bugs.mysql.com/bug.php?id=95787 [13:20:58] note we could setup a cronjob to restart mysql at a convenient time every week- not because it would be a great solution, but because we shoulf probably focus on an event-less alternative rather than a patch [13:21:51] yep [13:40:43] while playing with my abstraction, I noticed one thing [13:40:54] config.get_db_config(section='wikitech', dc='eqiad') [13:41:07] {'ro_reason': 'Maintenance on wikitech T229657 ', 'min_replicas': 0, 'readonly': False, 'master': 'db1133'} [13:41:08] T229657: Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC - https://phabricator.wikimedia.org/T229657 [13:41:24] ^we should probably reset the read only reason to something generic [13:41:49] I didn't know it stick after changing it to rw [13:41:51] interesting [13:42:02] I think it makes sense [13:42:06] it is not a huge issue [13:42:14] Also, I am not sure how to reset only that part without actually causing the RO to go thru [13:42:16] as for maintenance we would change it [13:42:21] oh, interesting [13:42:35] I don't think it is even possible [13:42:45] my only fear is that if it goes read only accidentally [13:42:46] maybe with edit section [13:42:47] let me see [13:42:58] e.g. master goes down [13:43:06] it may show a missleading reason [13:43:56] so this was all FYI, as I stumbled into it by chance [13:44:41] 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) I just realised we do have enabled TokuDB there, with a pretty big buffer configured 30GB. We don't have many TokuDB tables (as most of them are `FEDERATED`): ` +-----------------------+--------+---------... [13:45:01] Yeah, I can change the message with the edit [13:45:03] Let me change it [13:45:52] changed: [13:45:52] "readonly": false, [13:45:53] "ro_reason": "Please try again in a few minutes " [13:46:02] that's the message we had in the past [13:46:19] I can see it! config.get_db_config(section='wikitech', dc='eqiad') [13:46:24] {'ro_reason': 'Please try again in a few minutes ', 'min_replicas': 0, 'readonly': False, 'master': 'db1133'} [13:48:50] for example, I think wikis could got for a few seconds in read only if lagging, although not sure in that case they show that message to the api [13:50:43] yeah, don't know, maybe [13:51:27] 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Agusbou2015) Will enwiki the only wiki affected to this failover? [13:52:37] 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui) >>! In T231403#5468025, @Agusbou2015 wrote: > Will enwiki the only wiki affected to this failover? enwiki w... [14:16:45] I am going to put up es2001,2,3 mysql for production testing [14:26:23] ok! [14:45:27] 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10jcrespo) Those are the main tables where the historic metrics are stored- they are actually quite large. Most should disappear, but we cannot just convert them as they compress quite efficiently well compared to Inno... [19:26:18] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10WDoranWMF) [19:26:20] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10WDoranWMF) [20:25:37] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10eprodromou) @Anomie ISTM that we might get some value out of doing separate queries if the results are cached separately? Pseudo-code wise, ` if (cac... [20:33:01] 10DBA, 10Cloud-VPS, 10Data-Services: Queries of commonswiki_p.filearchive for fa_sha1 are slow - https://phabricator.wikimedia.org/T71088 (10Don-vip) [20:34:29] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10eprodromou) Speaking of caches, kind of off-topic for this ticket, but... At save time, we might be able to manipulate the cache directly, and thus s... [20:45:29] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10eprodromou) @Anomie I might be reading the queries wrong, but... is there a reason that you're using a limit of 100 instead of the page size of... [20:57:40] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10eprodromou) Last couple of points, and then I promise I'll stop commenting. - The initial scale of the API calls using these queries will be OOM 10... [20:58:17] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10Anomie) Hmm. Your comments made me realize I did counts of unique anon editors and unique bots, not count of edits by anon editors and by bots. Grr.... [21:00:21] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10Anomie) The 100 is arbitrary. This task didn't specify a page size. [21:06:43] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose Count Queries - https://phabricator.wikimedia.org/T231598 (10eprodromou) I claim full credit for debugging your queries that I neither read carefully nor commented on. You're welcome! [21:16:54] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10eprodromou) Cool. I wanted to comment here on some real-world optimization issues for these queries. - There's a fixed page size (20) for the...