[01:19:36] 10DBA, 10Community-Tech, 10Stewards-and-global-tools (Temporary-UserRights): Expired user groups not added to user_former_groups table - https://phabricator.wikimedia.org/T177404#3657453 (10TTO) The only solution I can think of, other than the scheduled task or jobqueue job proposed in T176754, would be to a... [01:35:09] 10DBA, 10Community-Tech, 10Stewards-and-global-tools (Temporary-UserRights): Expired user groups not added to user_former_groups table - https://phabricator.wikimedia.org/T177404#3657453 (10Legoktm) >>! In T177404#3659436, @TTO wrote: > That might be overkill though - wouldn't it just be better to alter the... [05:40:49] 10Blocked-on-schema-change, 10DBA, 10Readers-Community-Engagement, 10Community-Liaisons (Oct-Dec 2017): Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3659530 (10Marostegui) Thank you @CKoerner_WMF!! We'll update this... [05:42:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3659531 (10Marostegui) a:05MarkTraceur>03Marostegui As per: T176883#3658963 we will be upgrading the master on Wednesday 11th at 6:00UTC [05:44:26] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659536 (10Marostegui) >>! In T153033#3658843, @demon wrote: >>>! In T153033#3656161, @Marostegui wrote: >> Just to be clear, you are talking about dbstore1002/db1047? >> We also have to keep in mind that there a... [05:46:43] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659538 (10Marostegui) [05:47:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3638566 (10Marostegui) s7 master in codfw, db2029 finished the optimize, with replication. [05:47:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659540 (10Marostegui) [05:48:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3638568 (10Marostegui) [05:52:51] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659545 (10Marostegui) [05:53:26] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3638622 (10Marostegui) [06:11:35] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659552 (10Marostegui) [06:24:25] any reason why labsdb1009 has replication stopped on all its threads? [06:28:13] 10DBA, 10Patch-For-Review: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3659572 (10Marostegui) [06:52:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659593 (10Marostegui) [07:16:01] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659635 (10Marostegui) I have checked the tables across the masters: s1: has data on `enwiki` s2: has data only on: `nlwiki` s3: has data only on: ``` frwikisource incubatorwiki itwikivoyage sewikimedia tawiki... [07:48:09] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659653 (10demon) Testwiki we can drop for sure. So that just leaves 7 total wikis with viable data. Farrrrrrrr better. [07:49:19] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3564980 (10Legoktm) This ticket has a lot of comments so I might have missed it, but has someone verified that putting Commons into read only mode won't affe... [07:49:37] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3659659 (10Marostegui) The procedure I had in mind to upgrade the master: - Disable all replication alerts on s4. - Set commonswiki on read-only by merging:... [07:50:35] 10Blocked-on-schema-change, 10DBA, 10Readers-Community-Engagement, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review: Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3640012 (10Legoktm) Since this ticket is abo... [07:50:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3659662 (10Marostegui) >>! In T168661#3659656, @Legoktm wrote: > This ticket has a lot of comments so I might have missed it, but has someone verified that p... [07:52:45] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659668 (10Marostegui) Thanks @demon! I will exclude testwiki from the list of wikis the tables need to be imported from [07:54:22] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3659669 (10Legoktm) >>! In T168661#3659662, @Marostegui wrote: >>>! In T168661#3659656, @Legoktm wrote: >> This ticket has a lot of comments so I might have... [07:55:15] so s5 got fixed a bit [07:55:19] on dbstore1001 [07:55:24] now lag is going down [07:55:30] \o/ [07:55:37] we have to setup s4 [07:55:46] btw, you still working with labsdb1009? I saw replication is stopped in al lthe threads [07:55:52] I didn't start it as you might be doing something [07:55:53] and see why s4 was missbehaving [07:55:59] mm [07:56:25] did I forget to start replication or did it crash? [07:56:47] I didn't see any crashes [07:57:39] InnoDB: Warning: difficult to find free blocks in the buffer pool (322 search iterations)! [07:57:54] where's that? the last entry I am seeing is from july [07:58:09] not the error log is no longer on file [07:58:12] *note [07:58:17] but on systemd [07:58:23] shit! i forgot :) [07:58:44] so it is possible it crashed [07:59:15] I think it didn't [07:59:28] but the health doesn't seem very good to me [08:00:19] I am going to point back to labsdb1010 [08:00:28] ok [08:00:30] and please delay your maintentnace [08:00:33] on 9 [08:01:00] yeah, I am not touching 1009 :) [08:06:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659686 (10Marostegui) [08:09:00] do your alters now on 1009, it is depooled https://tools.wmflabs.org/replag/ [08:09:27] you sure? i can wait eh [08:09:30] until you have debugged it [08:09:37] it is ok [08:09:49] we have to break 1009 to see if it happens again [08:09:53] ok :-) [08:09:58] I will start the alters then! [08:10:22] I have analytics pointing to 1010 [08:10:27] awesome [08:10:32] we'll see how it goes [08:10:45] maybe there was some heavy querying or something [08:13:00] alters started [08:40:28] 10DBA, 10Patch-For-Review: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3659736 (10Marostegui) [08:45:03] jynus: https://phabricator.wikimedia.org/T151717#3633466 What do you think? [09:01:55] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 2 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3659854 (10jcrespo) > The data in user_groups isn't incorrect, there just happens to be a new column of relevant data.... [09:03:42] I asked you to extrapolate to the largest table possible [09:03:52] it seems commons or ruwiki have large wikidata usage [09:04:01] not as much enwiki [09:04:24] you talking about 1009? [09:04:35] answering to hoo^ [09:04:35] oh sorry [09:04:37] jynus: Can do [09:04:46] yeah,. missed his line in between the wikibugs, sorry! [09:05:09] is there wikidata team involvement on recentchanges coming from wikidata? [09:06:45] Well, we are working on it high priority now [09:07:02] well, mostly I do [09:07:13] 10Blocked-on-schema-change, 10DBA, 10Readers-Community-Engagement, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review: Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3659870 (10jcrespo) Agreeing with that^ [09:08:14] marostegui: there are some pending schema changes on commonswiki [09:08:35] we should check if there are things we should do at the same time [09:08:57] e.g. I think there is an index pending change, that could be done in any order (it is backwards and forward compatible) [09:09:14] probably not needed at the same time, but we should check pending schema changes [09:09:14] i will double check [09:09:20] I can do that, too [09:09:26] just communicating it [09:09:31] sure, it makes sense :) [09:09:38] so we do not "waste" a read only [09:09:51] specially for metadata-changes on image [09:09:56] *metadata-locking [09:10:16] should we have prepared a master failover, just in case? [09:10:52] yeah, we should at least select a candidate host [09:10:53] just in case [09:13:56] alter on dbstore1002 is 75% complete [09:14:11] maybe it will finish this evening [09:14:33] I will try to make dbstore1001 work, and reenable the lagged replication [09:16:24] can't wait to see dbstore1002 getting more space back [09:17:58] ok, so we didn't retrieve 1TB [09:17:58] \o/ [09:18:07] it was 400 Gb [09:18:13] the rest was the backup rotation [09:18:31] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3659885 (10hoo) On `elwiki` we saw the number of statement usage to be about 5.3 times the number of all + other usage... [09:18:32] marostegui: do you have 5 minutes to check my start s4 replication command? [09:18:32] jynus: https://phabricator.wikimedia.org/T151717#3659885 [09:18:47] jynus: sure! [09:19:01] I've put it on etherpad [09:19:25] I need this time really several eyes [09:19:39] ok [09:19:40] let me see [09:21:25] jynus: looks good to me! [09:21:33] ok, doing [09:21:48] oh, do I need to setup the filtering, or is it taken from configuration? [09:21:54] I will check it before start [09:22:09] it should be taken from config [09:22:36] it doesn't? [09:22:50] alt least it doesn't show on show slave status [09:22:57] I will aply it manualy just in case [09:23:04] Replicate_Wild_Do_Table: %wik%.%,heartbeat.% [09:23:06] i do see it [09:23:10] oh sorry [09:23:11] 1001 [09:23:22] yeah it is not there [09:23:30] i would have assumed it would have been taken from config :| [09:23:39] probably only on start [09:23:44] it has filter.s4 [09:23:52] but if it doesn't exist... [09:24:04] I almost prefer like this, not trying to be too clever [09:24:14] it is not a big deal anyway [09:25:24] ok, starting io thread [09:25:46] looking ok so far [09:26:05] so far so good [09:26:17] but replication lag going up [09:27:20] I think it is the globalimagelinks table [09:28:29] now 20 minutes to stop replication yay! [09:28:29] yeah lag is going up quite quickly indeed [09:30:37] at least s5 is responding nicely to treatment [09:31:16] -0.1 weeks in a day, whatever that means [09:35:17] to treatment haha [09:40:16] 10DBA, 10Community-Tech, 10Stewards-and-global-tools (Temporary-UserRights): Expired user groups not added to user_former_groups table - https://phabricator.wikimedia.org/T177404#3659923 (10MarcoAurelio) > whenever the user makes an edit, instead of waiting for someone to change their groups That sounds als... [09:45:05] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3659943 (10Marostegui) [09:49:26] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3659951 (10jcrespo) I am a bit lost with the estimation- is that realistic, is the number of usages more or less right... [09:56:56] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3659958 (10hoo) > I am a bit lost with the estimation- is that realistic, is the number of usages more or less right w... [09:57:33] jynus: FYI: I also picked up https://phabricator.wikimedia.org/T173196 yesterday [09:57:48] that should reduce the number of rows on commons significantly and make the table stable there [09:57:57] also it will reduce the number of addusage jobs there [09:58:01] significantly [10:00:21] I do not understand that task, but based on "will reduce the number of addusage jobs there" that sounds good :-) [10:03:54] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Test reliability of RAID configuration/database hosts on single disk failure - https://phabricator.wikimedia.org/T174054#3659978 (10Marostegui) db1076 has been depooled. @Cmjohnson ping me when you are online so I can start mydumper to simulate reads, a... [10:05:03] 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 2 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3659995 (10MarcoAurelio) [10:07:25] jynus: https://gerrit.wikimedia.org/r/382414 so this is ok with you? I would like to put it out today, so that we can start collecting the usages over the weekend [10:09:32] I'm not a fan of running refresh links over the whole wiki… maybe I have to see about writing a maintenance script that just updates our usage data [10:09:37] yes [10:09:45] specially important on s3 [10:10:05] serialize a bit deployment, remember the issues with 1-job-per wiki [10:10:06] or we just let it "naturally" run in [10:10:14] we used to have [10:10:27] scales on all replica sets except s3 (900 wikis) [10:10:39] busy now [10:11:00] Thanks :) A +1 would be nice… will put it up for SWAT later [12:46:17] 10DBA, 10Cloud-Services: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3611967 (10Marostegui) I saw this database was created on today's SWAT - going to sanitize it on the sanitarium hosts before handing it over to #cloud-services-team once SWAT is done [12:46:27] 10DBA, 10Cloud-Services: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3660492 (10Marostegui) p:05Triage>03Normal [12:49:10] 10DBA, 10Operations, 10procurement: Purchase testing backups hosts (2 hosts in total) in eqiad - https://phabricator.wikimedia.org/T177488#3660510 (10Marostegui) [12:55:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3660547 (10Marostegui) [12:56:21] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3642557 (10Marostegui) [13:48:30] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3660699 (10hoo) Table stats on `trwiki` pre-deploy: ``` No eu_aspect 24 L.en 460 L.tr 130725 O 508625... [14:41:23] T176043 broke replication on labs [14:41:23] T176043: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043 [14:41:52] the usual breakage that the database exists if followed by another one [14:41:58] I think Amir1 had to run the script several times, right? [14:42:22] yeah [14:42:30] it was a complete mess [14:43:19] Why was a wiki created in SWAT? [14:45:00] according to the Deployments page, that was a separate window from SWAT [14:45:18] https://wikitech.wikimedia.org/wiki/Deployments#Thursday.2C.C2.A0October.C2.A005 [14:45:48] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3660921 (10Andrew) Is there anything I can do to nudge this along, short of 'clone Jaime'? [14:47:42] Reedy: yes, the wiki had it's own window [14:48:06] labs fixed [14:48:38] Thank marostegui [14:48:41] *thanks [14:51:09] Amir1: What went wrong the first time of running addWiki? [14:52:20] dbstore1002 :s3 seems stuck creating indexes for amwikimedia [14:53:16] jynus: it happened last time a new wiki was created, and all of a sudden it went thru. There was also a big alter table running at the time, so maybe it will eventually go thru [14:53:17] I think is tokudb, where creating indexes is a very expensive operation [14:53:31] or it gets db-blocked [14:53:37] 10Blocked-on-schema-change, 10DBA, 10Readers-Community-Engagement, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review: Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3660941 (10CKoerner_WMF) Not a problem. I ju... [14:53:40] not a fan [14:59:15] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3660988 (10jcrespo) I think the first thing is to amend the description so that cloud (in particular), or anyone else agrees with the... [15:02:33] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3661002 (10Papaul) Disk wipe in progress . [15:17:02] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661093 (10Matthewrbowker) [15:19:39] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661093 (10Marostegui) It has been fixed already, it should be slowly recovering. [15:23:22] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661093 (10jcrespo) As a reminder, new replicas are more likely to be up to date than old ones (see they have 0 lag)- this is not on purpose- but newer hardware and extra redundancy make easie... [15:31:00] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661190 (10Matthewrbowker) >>! In T177505#3661118, @Marostegui wrote: > It has been fixed already, it should be slowly recovering. Cool! >>! In T177505#3661148, @jcrespo wrote: > As a remind... [15:31:58] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661191 (10jcrespo) Thanks! [15:33:35] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Test reliability of RAID configuration/database hosts on single disk failure - https://phabricator.wikimedia.org/T174054#3661198 (10Marostegui) So @Cmjohnson and myself have done the following tests: db1076 (testing host) pulled out one disk while gene... [15:37:34] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Test reliability of RAID configuration/database hosts on single disk failure - https://phabricator.wikimedia.org/T174054#3661211 (10Marostegui) 05Open>03Resolved Let's call it resolved for now then. Thanks a lot for your help @Cmjohnson [15:40:09] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3661214 (10Papaul) switch port information asw-a2-codfw ge-6/0/9 [15:42:48] 10DBA, 10Operations, 10ops-eqiad: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) - https://phabricator.wikimedia.org/T163778#3661219 (10jcrespo) This is probably not fully decomissioned yet (dns, puppet, etc.), but I am going to try to remov... [15:49:19] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Test reliability of RAID configuration/database hosts on single disk failure - https://phabricator.wikimedia.org/T174054#3661245 (10jcrespo) > I noticed that the disk shipped to db1076 to replace the failed one when it happened is bigger than the rest:... [16:03:04] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661348 (10Andrew) [16:08:17] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661387 (10bd808) p:05Triage>03Normal [16:09:19] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3351594 (10bd808) >>! In T167973#3660988, @jcrespo wrote: > I think the first thing is to amend the description so that cloud (in particular), or anyone... [16:12:38] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661408 (10jcrespo) > Is there anything I can do to nudge this along, short of 'clone Jaime'? Also don't underestimate the amount of time that you can h... [16:21:00] 10DBA, 10Operations, 10Availability (Multiple-active-datacenters), 10Performance-Team (Radar): Make apache/maintenance hosts TLS connections to mariadb work - https://phabricator.wikimedia.org/T175672#3661447 (10jcrespo) I would suggest to setup a proxysql instance to move this forward? maybe on terbium it... [16:24:31] 10DBA, 10Operations, 10ops-eqiad: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) - https://phabricator.wikimedia.org/T163778#3661467 (10jcrespo) Actually, I cannot do all the steps (network changes) without coordinating with DC ops. I do not... [16:51:44] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3661624 (10Cmjohnson) Case submitted with HP...Case ID 5323521514 [16:59:37] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3661662 (10Marostegui) Thank you! [17:11:57] jynus: follow-up question for you on https://phabricator.wikimedia.org/T177466#3661685 [17:18:38] already answered :-) [17:36:35] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661808 (10Matthewrbowker) Replag is now at 0 according to https://tools.wmflabs.org/replag/ [17:39:29] 10DBA, 10User-Matthewrbowker: Replag of S1, S3, and S5 are over 2.75 hours - https://phabricator.wikimedia.org/T177505#3661811 (10Marostegui) 05Open>03Resolved a:03Marostegui Closing this as the lag is gone as per replag Sorry for the inconveniences this might have caused! [20:20:56] [21:18:46] PROBLEM - MariaDB Slave SQL: s3 on dbstore1001 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1061, Errmsg: Error Duplicate key name eu_entity_id on query. Default database: amwikimedia. [Query snipped] [20:21:01] jynus: marostegui ^ it seems to be back [20:28:50] I know [20:28:54] don't worry [22:06:53] 10DBA, 10Operations, 10Availability (Multiple-active-datacenters), 10Performance-Team (Radar): Make apache/maintenance hosts TLS connections to mariadb work - https://phabricator.wikimedia.org/T175672#3662667 (10aaron) We discussed proxies in the last performance meeting and we're OK with that (it would cu...