[06:08:37] 10DBA, 10MediaWiki-Special-pages, 10Datacenter-Switchover-2018: Significant (17x) increase in time spent by updateSpecialPages.php script since datacenter switch over updating commons special pages - https://phabricator.wikimedia.org/T206592 (10Marostegui) 05Open>03Resolved [06:24:43] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) We shouldn't go for multi-source, we specifically discussed and bought hardware to avoid going for multi-source (T159423#4271324... [06:34:15] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Marostegui) So looks like it is all done per: https://github.com/Pathoschild/Wikimedia-contrib/issues/116#issuecomment-442907228 after @bd808 ran the script? [06:35:10] 10DBA, 10Operations, 10Patch-For-Review: Audit "misc" cluster hosts - https://phabricator.wikimedia.org/T210486 (10Marostegui) Ah right, the dbstore1003 and dbstore1005 are the new hosts that will replace dbstore1002 {T210478} [06:44:40] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [06:45:07] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [06:52:30] 10DBA, 10Operations, 10Patch-For-Review: Audit "misc" cluster hosts - https://phabricator.wikimedia.org/T210486 (10Joe) I would suggest you re-do your query excluding servers with `role::spare::system`, such as most of the cp* boxes and the conf* ones. Actually, it might make sense to put them in a separate... [07:25:44] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s2 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore2001 [] dbstore1002 [] db2095 [] db2091 [] db2088 [] db2063 []... [07:25:53] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s2 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore2001 [] dbstore1002 [] db2095 [] db2091 []... [07:29:07] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [07:29:21] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [07:34:17] marostegui: can you ask someone to invite me to #mediawiki_security? It seems I was banned [07:34:24] (good morning btw) [07:37:34] i have asked [07:39:16] tx [08:36:27] 10DBA, 10MediaWiki-Database, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10jcrespo) [08:39:25] is this good? https://phabricator.wikimedia.org/P7872 [08:41:28] it is expected [08:42:19] good [08:45:07] 10DBA, 10MediaWiki-Database, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Banyek) >>! In T210725#4785678, @jcrespo wrote: > > Regarding the high availability, I was thinking of setting up... [08:50:46] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) 05Open>03stalled I am stalling this until pc1007 is fixed as it arrived broken (T207258#4774505). Onc... [08:53:44] 10DBA, 10MediaWiki-Database, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) p:05Triage>03Normal [08:55:04] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) [09:22:17] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10MarcoAurelio) Note: we should maybe take a look at recently created wikis at #wiki-setup-create and check if they're also added to the meta_p.wiki table. Or compare its c... [09:23:28] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10MarcoAurelio) [10:35:06] 10DBA, 10Data-Services, 10Tool-Quentinv57's-tools, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343 (10MarcoAurelio) Filed {T210829}. [11:10:39] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [11:18:47] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Banyek) p:05Triage>03Normal [11:19:32] 10DBA, 10Cloud-Services, 10cloud-services-team, 10User-Banyek: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Banyek) a:03Banyek When the wiki is created please ping me and I'll do the sanitization [11:26:07] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10GTirloni) 05Open>03Resolved [11:39:28] we have an accesss request: https://phabricator.wikimedia.org/T208622#4786302 [12:07:37] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Performance-Team (Radar): Investigate solutions for MySQL connection pooling - https://phabricator.wikimedia.org/T196378 (10jcrespo) a:05jcrespo>03None We are on discussion to see when #DBAs can move this forward (pending testing proxysql2 pack... [12:08:06] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10jcrespo) a:05jcrespo>03None [12:09:26] 10DBA, 10Goal: Design and prepare infrastructure for database binary backups - https://phabricator.wikimedia.org/T206203 (10jcrespo) [12:09:28] 10DBA: Research options for producing binary backups (lvm snapshots, cold backups, mariabackup) - https://phabricator.wikimedia.org/T206204 (10jcrespo) 05Open>03Resolved Resolved, it is most likely going to be mariabackup/xtrabackup, it just has not been proved on a production load. Followup on T210292 [12:46:52] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) [13:06:36] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10Banyek) @Marostegui I think we should close this task, as the replication is good, and I doubt if we'll replace that BBU befure decommisioning. [13:28:42] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10jcrespo) 05Open>03declined Technically the alerts went away after the restart, lets decline it because we know it is not in a good state and it is likely to reappear, but I agr... [13:31:56] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) >>! In T210478#4788172, @Marostegui wrote: > We shouldn't go for multi-source, we specifically discussed and bought hardware to avoi... [13:33:02] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10Marostegui) Agreed with all you guys said Further, we not only not invest in old hardware but these hosts should go away once we've got the final backups hosts in place [13:46:34] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10jcrespo) > We can use the new ones We don't have proxies for this purpose, the proxies you worked on are assigned to misc and should stay f... [13:48:45] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) On top of what Jaime wrote, which is basically the same thing I was writing...we also have to consider the fact that these hosts... [14:00:26] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) with "We can use the new ones" I was wondering that we could (not should) use them beside the current role of the new proxies - tha... [14:00:57] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) Thanks a lot for all the inputs, I'd say that we don't need proxies for the moment, we'll probably just need some automation around... [14:17:50] I will leave soon(ish) for the weekend, can I do anything for you before that? Tomorrow I'll travel to the countryside in the morning, and I'll drive back sunday after lunch FYI. (I'll have my phone) [14:18:00] Jaime, have a wonderful vacation!!! [14:20:13] banyek: have a nice weekend! [14:20:35] thanks [14:20:37] * banyek waves [15:03:56] 10DBA, 10MediaWiki-Database, 10TechCom-RFC: RFC: Proposal to add wl_addedtimestamp attribute to the watchlist table - https://phabricator.wikimedia.org/T209773 (10D3r1ck01) > @D3r1ck01 is there community support for this idea? Do many people find this useful? Great question @daniel. I think we should find o... [15:38:36] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: MySQL metrics monitoring - https://phabricator.wikimedia.org/T143896 (10jcrespo) 05Open>03stalled [18:15:50] 10DBA, 10Analytics, 10Cloud-VPS, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) A possible solution, instead of ordering new hardware, would be to reuse one/two of the new Hadoop nodes racked in T207192 for this use case: they have 1... [18:18:02] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10bd808) [18:24:31] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) >>! In T210749#4789691, @elukey wrote: > A possible solution, instead of ordering new hardware, would be to reuse one/two of the new Hadoop nodes... [18:28:41] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) >>! In T210749#4789731, @Marostegui wrote: >>>! In T210749#4789691, @elukey wrote: >> A possible solution, instead of ordering new hardware, would be... [18:29:51] 10DBA, 10MediaWiki-Database, 10TechCom-RFC: RFC: Proposal to add wl_addedtimestamp attribute to the watchlist table - https://phabricator.wikimedia.org/T209773 (10daniel) @D3r1ck01 wrote > I don't know if a channel we can use but maybe others have ideas? There is no single channel to the community, since th... [18:30:57] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) 128GB + non SSDs disks will definitely not work for heavy queries :-( [19:34:48] Does MediaWiki have any db table where broken and double redirects are stored?