[00:01:37] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674309 (10EddieGP) Per comments given on the patch, it will need a +1 from #DBA to proceed. I'm moving this on your w... [00:11:00] 10DBA, 10Data-Services, 10XTools, 10cloud-services-team: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674325 (10MusikAnimal) Your comment was at 06:22 UTC on 9 October. Here's what I've got when grepping the logs... [00:16:19] 10DBA, 10Data-Services, 10XTools, 10cloud-services-team: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674332 (10MusikAnimal) Scratch that, the logs clearly say `s5003` and not `s51187` =P The cache has been cleare... [05:15:55] 10DBA, 10Operations, 10ops-codfw: db2038 two disks with predictive failure - https://phabricator.wikimedia.org/T177720#3674533 (10Marostegui) 05Open>03Resolved And all good now! Thanks a lot @Papaul ``` root@db2038:~# hpssacli controller all show config Smart Array P420i in Slot 0 (Embedded) (sn: 00... [05:21:31] 10DBA: dbstore2001:s3 crashed while backups were running - https://phabricator.wikimedia.org/T177908#3674537 (10jcrespo) [05:28:31] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674551 (10Legoktm) I'm worried that T168661#3663516 is still an issue, but I suppose we can just clean it up afterwards by running refreshLinks.php everywhere. [05:29:33] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674552 (10Marostegui) >>! In T168661#3674551, @Legoktm wrote: > I'm worried that T168661#3663516 is still an issue, but I suppose we can just clean it up af... [06:19:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674595 (10Marostegui) This maintenance is finished Read only time started at: 6:01 Read only time finished at: 6:14 Total 13 minutes The master has now t... [06:20:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674598 (10Marostegui) [06:20:17] 10DBA, 10Readers-Community-Engagement, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review: Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3674596 (10Marostegui) 05Open>03Resolved This maintenance is finished... [06:23:45] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674600 (10Marostegui) [06:24:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3569016 (10Marostegui) This is all done - I am going to check again (T168661#3576396) across all the fleet to make sure this is done everywhere before closin... [06:26:09] feel free to start the recentchanges purge again on db1068 [06:27:13] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674603 (10jcrespo) That requires an order-by, a limit on a loop, and a waitfor replica on every loop step. [06:27:20] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3674607 (10Marostegui) [06:27:23] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3674604 (10Marostegui) 05Open>03Resolved [06:34:32] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674608 (10TTO) >>! In T176754#3674603, @jcrespo wrote: > That requires an order-by, a limit on a loop, and a waitfor... [06:37:25] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674609 (10Marostegui) >>! In T176754#3674608, @TTO wrote: >>>! In T176754#3674603, @jcrespo wrote: >> That requires a... [06:41:22] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674610 (10jcrespo) > will always be very small And recentchanges from wikidata will be a small percentage... https:/... [06:52:14] I will take a small break, but let me know what you think of restarting the purges on commons rcs [06:52:36] ah, i already mentioned it [06:52:42] ˜/marostegui 8:26> feel free to start the recentchanges purge again on db1068 [06:52:45] :-) [06:54:08] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674614 (10TTO) >>! In T176754#3674610, @jcrespo wrote: >> will always be very small > > And recentchanges from wikid... [06:55:15] sorry [06:55:18] I missed that [06:55:32] no worries, easy to miss things with the bots - happens to me all the time! [06:57:28] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3674617 (10jcrespo) Look, I ask for a loop and probably a SELECT... FOR UPDATE, that should be no more than a 3 line c... [07:23:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674658 (10jcrespo) @Legoktm will you want to file a ticket for something like "smoother read only" for other projects when commons or wikidata are down/ro o... [07:48:39] can db2044 reenable alerting? [07:49:23] db2044? [07:49:29] can't remember what happened with it [07:49:32] ah [07:49:35] I think I remember [07:49:40] it was the host that crashed? [07:49:45] becase RAID? [07:49:47] oh, speaking of alerting, going to enable db2080 [07:49:51] oh yes [07:49:52] was board changed? [07:49:53] that one [07:49:53] true [07:49:57] yes yes, you are totally right [07:50:26] T174764 [07:50:27] T174764: db2044 HW RAID failure - https://phabricator.wikimedia.org/T174764 [07:50:33] I will reenable it [07:50:34] yep, totally fine to enable them [07:50:36] as it is pooled [07:50:49] I also didn't remember why it was disabled [07:50:56] db2080? [07:50:58] enable? [07:51:00] puppet? [07:51:15] sorry, enable notifications [07:51:24] by removing the line from db2080.yaml :) [07:51:30] ok [07:53:16] we need to make sure alerting is working just before pooling them [07:53:32] otherwise we could miss issues while active [07:54:06] yeah, totally agree [07:54:09] i missed it [07:54:37] not your fault [07:54:55] again, I mean we as in royal "we" [07:54:58] :-D [07:55:09] haha [07:56:27] not sure if dbstore2001 is recovering lag, although with backups ongoing, not something worring [07:57:07] sadly, like dbstore1001, it lags because of backup load [08:01:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3674694 (10Marostegui) 05Open>03Resolved It is everywhere, so closing this. Thanks everyone involved in making possible to close this task! [08:07:56] 10DBA, 10Data-Services, 10XTools, 10cloud-services-team: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674700 (10Marostegui) Cool - let me know how it goes. I didn't want to add exceptions to two users, if we can a... [08:11:23] 10DBA: dbstore2001:s3 crashed while backups were running - https://phabricator.wikimedia.org/T177908#3674701 (10Marostegui) Let's do a transfer from dbstore2002 once the backups are finished [08:15:35] 10DBA: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3674703 (10Marostegui) [08:17:04] 10DBA: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3674729 (10Marostegui) p:05Triage>03Normal db1038 can now be decommissioned. db1072 has been pooled as vslow and dump service on s3 with db1038's data Let's not touch it until Monday though, just to make sure db1072 works fine. Next week... [08:17:59] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3674735 (10Marostegui) [08:18:01] 10DBA: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3674734 (10Marostegui) [08:19:30] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266773 (10Marostegui) [08:19:41] 10DBA, 10Operations, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3674736 (10Marostegui) [08:19:48] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3674740 (10Marostegui) [08:19:50] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1037 - https://phabricator.wikimedia.org/T174902#3674739 (10Marostegui) [08:21:34] 10DBA: dbstore2001:s3 crashed while backups were running - https://phabricator.wikimedia.org/T177908#3674755 (10jcrespo) Note mysql and replication is running- we could just drop and reload the "asserted" table. [08:22:38] 10DBA: dbstore2001:s3 crashed while backups were running - https://phabricator.wikimedia.org/T177908#3674756 (10Marostegui) >>! In T177908#3674755, @jcrespo wrote: > Note mysql and replication is running- we could just drop and reload the "asserted" table. Sure, we can do that too, if there is only one table th... [08:23:29] should we make db1072 the new ROW s3 replica? [08:24:14] yeah, good idea [08:24:45] you pooled db1103 on s3 [08:25:11] i did [08:25:22] it has a bad comment [08:25:47] ah yeah, i see [08:25:53] let me remove it, actually, I will depool it [08:25:59] but we do not need a special slave on s3 [08:26:02] there is no need for it really [08:26:13] we need that server for misc [08:26:13] yeah exactly, I will depool it [08:26:29] I mean [08:26:33] not that one exactly [08:26:59] nah, db1103 is there because it has db1035's data, so it is there so we can checksum its data and then removing it from s3 [08:27:05] yes [08:27:10] ok [08:27:19] let me make it clear on the comment [08:27:24] then the comment confused me [08:27:33] nota big deal [08:27:54] I thought you were setting up a new large server there [08:28:46] https://gerrit.wikimedia.org/r/383516  [08:28:47] just comment the source, nothing else [08:28:51] is it clearer now? [08:29:02] yep [08:29:06] no need to deploy [08:29:06] \o/ [08:29:15] do it when you do other [08:29:28] I just got confused with the role comment [08:29:30] yeah :) [08:29:32] sorry to bother you [08:29:35] no no [08:29:44] when I read it again after 2 weeks, it was also confusing to me [08:29:46] XD [08:29:48] yeah [08:29:54] at least you did it [08:29:59] yeah haha [08:30:02] i had some context [08:30:02] I was like, what is this? [08:30:12] and I knew you mentioned [08:30:18] but I didn't remember [08:30:32] now I do [08:30:53] Is Manuel about to partition 800 tables? XD [08:31:23] hahaha [08:31:41] actually, I am going to deploy, so I can checksum it against db1038, as they might have the same data [08:31:57] it complains about the space at the end, I think [08:32:02] yeah [08:32:05] just amending [08:53:19] 10DBA: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3674789 (10Marostegui) [08:53:20] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3674790 (10Marostegui) [09:13:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3674861 (10Marostegui) [09:33:44] hi, I'm working on checking smart attributes from disks, is there a couple of db hosts in codfw I could use to trial it? bonus points if they have different raid controllers [09:36:44] godog: which are the implications? :) [09:36:49] ie: you are not going to crash them, right? [09:38:11] db2044 (had its mainboard recently replaced, so it will be good if it is actually fine - HP host) and db2079 (a new host with a perc controller) [09:38:13] marostegui: not on purpose no! [09:38:39] the net side effect is that smartmontools is installed, smartd running [09:38:56] and smartctl ran on all physical disks [09:40:38] marostegui: ok thanks! I'll add you and jynus to the code review [09:40:51] cool! [09:44:19] "smart attributes from disks"? is it to predict drive failure or in general? [09:50:32] manuel: the backup finished before the crash, and it was using gtid, so I am just going to delete the table and reload it [09:50:42] (dbstore2001:s3) [09:51:00] ok thanks :) [10:32:06] 10DBA: dbstore2001:s3 crashed while backups were running - https://phabricator.wikimedia.org/T177908#3675002 (10jcrespo) 05Open>03Resolved a:03jcrespo Given the crash happened after the backup finished, it used GTID and there is anoteher copy of s3 on dbstore2002, I have just dropped and reloaded the table... [10:32:38] \o/ [10:35:10] I do not really think tokudb is a good engine for reads [10:35:22] it struggles to run a count(*) [10:35:42] and yes, cool that it writes quickly, but if later we cannot read from it [10:35:54] haha yeah [10:36:08] and compresses good, but you cannot read either :p [10:36:11] maybe it got better on stretch/10.1? [10:36:50] innodb seems to have optimized select count(*) for innodb on 10.1 already [10:37:15] it is not instant, but it takes 10 seconds to count 40 Million rows [10:37:43] (I assume they are not technically counted, just some witty caching on the PK tree) [10:38:05] not sure if we will be able to test tokudb on 10.1, hopefully not :) [10:38:25] we will [10:38:34] (eventlogging) [10:38:43] oh yes, event logging, indeed [10:38:50] speakign of which, what happened with the new servers? [10:38:59] Oh , I think I saw an update yesterday [10:39:01] there is some pending work [10:39:06] still being racked and dns and things no? [10:39:08] dns or something [10:39:15] it is mostly done [10:39:50] I am thinking on purging ruwiki on parallel [10:39:57] as it will be taking so much time [10:40:14] how many need deletion on ruwiki? [10:40:15] it is only slightly smaller than commmons [10:40:28] 37M rows total, counting now the ones to be purged [10:49:52] 36207647 to delete [10:50:10] it is both less (half than commons), but much more in proportion [10:50:36] yeah, not bad, so it will take 15 hours or so? [10:50:45] we will see [10:51:41] 10DBA, 10MediaWiki-Watchlist, 10Wikidata: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) - https://phabricator.wikimedia.org/T177772#3675017 (10jcrespo) ruwiki results are more extreme: ``` root@db2076[ruwiki]> SELECT count(*) FROM r... [11:16:40] s6 has much less write load, at least at this time, so for now it is not lagging [11:37:42] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3675147 (10Marostegui) [11:42:20] 10DBA, 10Analytics, 10Analytics-EventLogging: Potentially decrease db1046's InnoDB buffer pool - https://phabricator.wikimedia.org/T125829#1998057 (10Marostegui) Is this still worth keeping considering the fact that we are going to replace this host soon anyways? [11:43:23] 10DBA, 10Analytics, 10Analytics-EventLogging: Potentially decrease db1046's InnoDB buffer pool - https://phabricator.wikimedia.org/T125829#1998057 (10elukey) I would not touch db1046 from now if possible :) [11:45:47] 10DBA: include primary key field as secondary index suffix - https://phabricator.wikimedia.org/T105440#3675177 (10Marostegui) 05Open>03declined I am going to close this as I don't think we are planning to work on this anyways. I don't think we have been suffering this issue in the last year, and specially w... [11:52:58] 10DBA, 10DC-Ops, 10Packaging, 10Patch-For-Review, 10Wikimedia-Incident: Change oom_adj for dedicated mysql server processes - https://phabricator.wikimedia.org/T172494#3500171 (10Marostegui) After those patches, is this considered done? [11:54:41] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3675187 (10Marostegui) [12:41:50] jynus: Can you check whether the write rate on cawiki (s7) is ok currently? [12:42:03] hoo: anything you've done could be affecting s4? [12:42:20] there is a massive spike on both s4 and s7 [12:42:24] https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?panelId=7&fullscreen&orgId=1&from=1507724951138&to=1507725535221&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=s4&var-shard=s7&var-role=All [12:42:27] I'm probably s7 [12:42:30] crap :/ [12:42:36] but s4 is definitely not me [12:43:09] they both happened at exactly the same time [12:43:11] can we revert? [12:43:16] sure [12:43:56] hang on [12:43:58] it looks stable now [12:44:03] as in it has decreased [12:44:36] Revert is on the way [12:44:43] for cawiki only [12:44:46] cewiki looks sane [12:45:50] marostegui: Are we good now? In that case, I'll permanently revert cawiki [12:46:04] let's revert cawiki yeah, just in case [12:46:44] Guess we have to… I wonder why it escalated *this quickly* [12:47:03] s4 might be the recentchanges purge I think [12:47:42] I am not 100% sure it might be your thing, but let's not guess :-) [12:48:25] jynus: going to stop the purges, just for now on s4 master [12:50:26] ok, so rows written graph for s7 and s4 was the recentchanges thing [12:50:27] I believe [12:50:34] https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?panelId=7&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=s4&var-shard=s7&var-role=All&from=now-1h&to=now [12:50:40] because you haven't still deployed, right? [12:50:56] I am writing on s6, not s7 [12:51:05] jynus: i know [12:51:25] look at the above graph, and I believe hoo has not fully reverted yet, right? [12:51:43] marostegui: I have reverted… but the change is still local on tin [12:51:52] right, i meant it is not deployed [12:52:11] it is… and I can see that no new usage are coming in on cawiki [12:52:11] don't look at the aggregated [12:52:24] look at the master individual query tiles [12:53:41] I have started the purge again on s4 [12:54:10] nooo [12:54:28] ? [12:54:33] ok, it worked, nevermind [12:55:09] I thought it didn't append, but overwrote archived rows [12:55:14] it still appends, so no issue [12:55:23] Is s7 fine again? [12:55:34] hoo: yeah, looks so [12:55:46] what was the change? [12:55:57] hoo: i am not completely sure it was your change though [12:55:57] jynus: Statement usage tracking for cawiki [12:56:07] it probably was [12:56:11] xdd [12:56:18] We added thousands of rows in just seconds [12:56:26] no idea how that could escalate so quickly [12:58:36] but this might be enough to at least understand what's going on on cawiki [12:58:40] will put on the ticket in a bit [12:59:28] sure, thanks [12:59:45] keep in mind that tomorrow is a public holiday in spain, so jaime and myself won't be online (just saying) [13:02:31] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3675355 (10Marostegui) [13:03:45] 10DBA, 10Analytics, 10Analytics-EventLogging: Potentially decrease db1046's InnoDB buffer pool - https://phabricator.wikimedia.org/T125829#3675361 (10jcrespo) 05Open>03declined [13:04:33] marostegui: That's fine… I'll need to carefully analyze this first anyway, before we can even think about trying cawiki again [13:04:44] Sorry for the mess and thanks for the support [13:05:17] no worries! [13:06:46] 10DBA: include primary key field as secondary index suffix - https://phabricator.wikimedia.org/T105440#3675484 (10jcrespo) This is actually quite relevant- it is the cause of many FORCE indexes, and we did it for special slaves. We are not going to "do it", but we should be aware of it because we could revert ma... [13:08:01] 10DBA, 10DC-Ops, 10Packaging, 10Patch-For-Review, 10Wikimedia-Incident: Change oom_adj for dedicated mysql server processes - https://phabricator.wikimedia.org/T172494#3675490 (10jcrespo) 05Open>03Resolved a:03jcrespo yes, but it should be increased for other packages, as it will not take real effe... [13:28:13] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3675563 (10hoo) Table sizes pre-deploy to `cawiki` and `cewiki`: ``` mysql:wikiadmin@db1079 [cawiki]> SHOW TABLE STAT... [14:02:53] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3675687 (10hoo) `cawiki` had (at the time statement usage were disable again) 90,430 statement usages from just 1,118... [14:06:47] the stretch 9.2 point update ships a new kernel (4.9.51), so about four months of 4.9.x bugfix backporting compared to 4.9.30. I'll install it on the db hosts already running stretch, that way the hosts which get rebooted for hw maintenance etc. can pick up the new kernel. ok with DBAs? [14:16:04] yes [14:32:38] ok [15:28:59] hello everybody, as fyi I am going to drop some el tables for https://phabricator.wikimedia.org/T171629#3674250 [15:29:09] from db1046/47/store1002 [15:29:15] I'll stop el_sync etc.. [15:29:23] \o/ [15:29:27] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3676023 (10jmatazzoni) In T171027#3673060, @Lydia_Pintscher w... [16:30:18] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3676228 (10Bawolff) >>! In T171027#3676023, @jmatazzoni wrote... [17:14:03] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3676386 (10Dzahn) https://gerrit.wikimedia.org/r/382495 [17:21:30] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3676424 (10jrbs) Mind if I assign you @Reedy ? [17:27:05] 10DBA, 10MediaWiki-Platform-Team, 10monitoring, 10Performance-Team (Radar): Improve database aplication performance monitoring visibility - https://phabricator.wikimedia.org/T177778#3676478 (10Gilles) [18:29:20] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3559677 (10Dereckson) @jrbs You can let it unassigned, so you've more opportunity for a swift deployment. [19:43:12] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3677195 (10Bawolff) > Well wikidata is almost certainly a con... [20:28:36] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3677449 (10Catrope) Thanks, I'll experiment with removing the... [22:29:45] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3600098 (10Alsee) >>! In T171027#3673060, @Lydia_Pintscher wr...