[01:13:29] 06Traffic, 06Infrastructure-Foundations, 06SRE: Console domain and property access request - https://phabricator.wikimedia.org/T381904#10507045 (10Scott_French) Thank you both! Great, the list in T381904#10502098 is consistent with what I have from when I did the correlation with Search Console properties i... [05:57:21] 06Traffic, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10507242 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs4009.ulsfo.wmnet with OS bookworm [06:42:13] 06Traffic, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10507254 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs4009.ulsfo.wmnet with OS bookworm completed: - lvs4009 (**WARN**) - Downtimed on... [07:21:40] 06Traffic, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10507303 (10Vgutierrez) [10:01:26] 06Traffic, 10conftool, 10Hiddenparma: Create a "trusted bot" stanza from requestctl we can use to bypass other rules - https://phabricator.wikimedia.org/T385149 (10Joe) 03NEW [10:01:39] 06Traffic, 10conftool, 10Hiddenparma: Create a "trusted bot" stanza from requestctl we can use to bypass other rules - https://phabricator.wikimedia.org/T385149#10507621 (10Joe) p:05Triage→03Medium [12:34:33] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10508147 (10KartikMistry) >>! In T383750#10503233, @LSobanski wrote: > @KartikMist... [14:39:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Check link from msw1-eqiad et-0/1/0 to msw2-eqiad et-0/1/0 - https://phabricator.wikimedia.org/T384708#10508615 (10Papaul) 05Open→03Resolved a:03Papaul We are not seeing any errors for the last 24 hours resolving this task fo... [15:13:06] o/ after the backports are finished, I'd like to roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1115056. Would that be okay? It already looked good last time I rolled out, just needed some updates to the headers the service was emitting (etag and cache-control) [15:48:49] hnowlan: sounds good [16:05:18] vgutierrez: thanks! Starting ~now [16:07:22] vgutierrez: congrats for the Liberica work :) [16:07:40] thx elukey <3 [16:10:03] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10509243 (10cmooney) 05Open→03Resolved a:03cmooney [16:16:50] okay, change looks good, impacts only testwiki as expected - I'm going to enable puppet. [16:24:29] hnowlan: nice :D [16:49:10] 10netops, 06Infrastructure-Foundations, 06SRE: Manage fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10509343 (10cmooney) [16:51:37] 10netops, 06Infrastructure-Foundations, 06SRE: Manage fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10509388 (10cmooney) 05Open→03Resolved This is now largely complete. We have decided to model the switch<->server links in Netbox (with dummy names 'PRIMARY_A' a... [17:00:05] RhinosF1: Thanks for the heads up! [17:00:36] brett: our user also says they are having trouble with Windows 11 too but microsoft guides imply it should be fine [17:00:54] I don't have a Windows machine to test on though [17:03:53] I'd be really surprised about that [17:04:22] brett: i was very surprised too [17:04:41] i can't test myself though as i don't have personal windows devices and AWB is windows only [17:05:46] https://learn.microsoft.com/en-us/windows/win32/secauthn/protocols-in-tls-ssl--schannel-ssp- ? [17:06:23] brett: that's what I looked at too [17:06:33] but Windows 11 AWB is saying no [17:06:49] O.o [17:06:50] Windows 10 definately based on that doesn't support TLS1.2 [17:07:59] > Support for TLS 1.3 was added to Secure Channel (schannel) for the GA releases of Windows 11 and Windows Server 2022.[63] [17:08:01] https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_1.3 [17:08:09] yup [17:08:26] so I'm not reading it wrong at least [17:08:28] RhinosF1: does the user have any sort of anti-virus stuff installed that might include a MITM proxy [17:08:36] cdanis: good question [17:08:44] they are getting https://cdn.discordapp.com/attachments/1334487775294263357/1334487775587598366/image.png?ex=679cb617&is=679b6497&hm=bb141e14cbd1856277fbcf2122670b8b3183b8ad85e64b1f32cbc52758356b4f& [17:09:29] :o [17:10:33] only windows kit i have is day job stuff so hard to verify it [17:10:41] it's the only complaint we've had though [17:10:45] I can VM it later [17:11:14] brett: cool, anything under miraheze.org or betaheze.org is TLS1.3 only [17:11:29] i can grant rights on meta.betaheze.org [17:39:43] hello traffic friends, back once again with an ATS Lua change [0] (this time a fairly minor one that fixes an edge case involving config handling). [17:39:43] any concerns / objections if I were to roll this out at some point this morning, potentially in the 18:00 UTC hour? [17:39:43] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1084247 [17:40:46] I'd plan to do the usual disable puppet -> pilot on a single cp-text host -> roll out technique [17:52:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509771 (10cmooney) [17:52:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509779 (10cmooney) [17:53:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509783 (10cmooney) [17:56:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509812 (10cmooney) [17:56:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509815 (10cmooney) [17:58:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10509830 (10cmooney) [18:06:00] brett: no proxy according to the user [18:06:36] It looks like AWB might need to be recompiled with a newer .NET and then it would only work on Win 11 [18:06:48] But ye current AWB doesn't work with TLS1.3 [18:07:05] Is my understanding without any testing of it [18:09:35] RhinosF1: Thanks so much for getting that info! I'll add it to the ticket [18:34:41] Do we have a timeline for WMF rollout of TLS 1.3? [18:34:47] Or more specifically, disabling 1.1/1.2? [18:35:22] brett: If AWB is responsible for a lot of that sort of traffic... I'm probably the person that's gonna have to fix it [18:39:27] I haven't heard of a timeline yet [18:39:47] Reedy: compiling it in a more recent version of .NET would be the start [18:39:58] I have no idea how bad the jump from 4.0 to 4.6 would be though [18:39:59] Yeah, need to work out what version that actually is [18:40:10] Reedy: 4.0 is current and min is 4.6 [18:40:10] we're not on 4.0 ;) [18:40:36] We're on 4.5 (at least) [18:41:39] >The Microsoft .NET Framework 4.6 is a highly compatible, in-place update to the Microsoft .NET Framework 4, Microsoft .NET Framework 4.5, Microsoft .NET Framework 4.5.1 and Microsoft .NET Framework 4.5.2. [18:41:41] how nice of them [18:42:49] Reedy: the latest version of AWB according to my user is showing as built in 4.0 [18:43:03] they may be wrong [18:43:08] But that's the info I'm getting [18:43:12] From where? [18:43:18] Oh [18:43:18] It needs to be 4.6 to support TLS1.3 [18:43:31] the .NET version we're showing... is the CLR(?) version from memory [18:43:51] Who is we? [18:43:57] we as in AWB [18:44:04] Do remember I do most of the AWB dev ;) [18:44:29] Yeah... Environment.Version which returns the CLR version [18:44:32] oh right [18:44:34] Nice [18:45:15] If it's highly compatible, in place, maybe it's possible to just recompile it for .NET4.6 then [18:45:24] And that'll make it work for Win11 [18:45:28] Not sure about Win10 [18:45:33] I think [18:45:40] If I understand the docs correctly Reedy [18:45:51] Yeah, depends how much of it is based on the underlying OS' libraries [18:46:04] https://learn.microsoft.com/en-us/dotnet/api/system.environment.version?view=net-5.0 [18:46:07] >For the .NET Framework Versions 4, 4.5, 4.5.1, and 4.5.2, the Environment.Version property returns a Version object whose string representation has the form 4.0.30319.xxxxx. For the .NET Framework 4.6 and later versions, and .NET Core versions before 3.0, it has the form 4.0.30319.42000. [18:46:24] I don't think a bump to 4.6 is an issue [18:46:26] >Windows 7 Service Pack 1, Windows 8, Windows 8.1, Windows Server 2008 R2 SP1, Windows Server 2008 Service Pack 2, Windows Server 2012, Windows Server 2012 R2, Windows Vista Service Pack 2 [18:46:58] But you look to be right... [18:46:59] >TLS 1.3 is supported starting in Windows 11 and Windows Server 2022. [18:47:13] Oh ok [18:47:15] >TLS 1.3 is supported starting in Windows 11 and Windows Server 2022. Enabling TLS 1.3 on earlier versions of Windows is not a safe system configuration. [18:47:16] That makes sense [18:47:39] Why is the version string 4.0 for all of them [18:47:43] That confused me [18:47:54] See the environment.version link above [18:48:09] brett: As a heads up... It seems moving to TLS 1.3 (only), which I know is down the line... Is going to potentially break access for basically anyone not on Windows 11 :D [18:48:25] Reedy: browsers work fine [18:48:43] I'd expect a lot more to moan if we'd broken complete windows 10 access [18:49:20] But this user claimed to be on Windows 11 with AWB and still having issues [18:49:28] TLS 1.3 is disabled by default on most browsers it seems (atm) [18:49:55] I don't think that's true, we haven't seen any drop in human traffic [18:50:48] Reedy: https://caniuse.com/tls1-3 [18:51:01] RhinosF1: Looking at edge on my windows 10 WMF laptop... 1.3 isn't enabled [18:51:05] marked experimental [18:51:07] and most traffic was TLS1.3 anyway [18:51:21] Reedy: hmm, can you access meta.miraheze.org [18:51:27] If you can, it's lying to you [18:53:07] Our data shows 98.4% of traffic was TLS1.3 in last 30 days (I only disabled 1.2 on Tuesday ) [18:53:42] >Your user agent supports TLS 1.2 and TLS 1.3, which are recommended protocol version at the moment. [18:53:52] I have no idea what that setting actually is [18:55:17] https://www.microsoft.com/en-us/security/blog/2020/08/20/taking-transport-layer-security-tls-to-the-next-level-with-tls-1-3/ [18:55:29] I guess by now they've (re)-implemented things in the browser? [18:55:37] >The Chromium-based Microsoft Edge does not use the Windows TLS stack and is configured independently using the Edge://flags dialog. [18:55:39] Yeah then [18:55:45] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10510057 (10BCornwall) @RhinosF1 was kind enough to do some investigation for AutoWikiBrowser/Windows. .NET... [18:55:49] >TLS 1.3 support will also be added to .NET beginning with version 5.0. [18:56:18] That link brett just put on the task... [18:56:19] >TLS 1.3 support will also be added to .NET beginning with version 5.0. [18:56:20] Ye [18:56:33] I guess it actually needs .NET 5 then [18:56:46] jeez, thanks windows [18:56:47] we do need 4.6.2 (at least) to get it on windows 11, but not 10 [18:57:03] I'll have to do some testing in anger :P [18:57:16] https://learn.microsoft.com/en-us/windows/win32/secauthn/protocols-in-tls-ssl--schannel-ssp-#tls-protocol-version-support [18:57:25] Okay [18:57:30] That makes some sense [18:57:44] But yes, thank you Microsoft, very helpful [18:58:06] But windows 11 is a free* upgrade!!! [18:58:10] Reedy: how difficult would a version of AWB with .NET 4.6.2 / .NET Core 5 be? [18:58:21] 4.6.2 shouldn't be difficult [18:58:21] Reedy: I spent all week upgrading work kit to Win 11 [18:58:24] 5.. I'm not sure :) [18:58:26] Free in money [18:58:28] Yes [18:58:28] This throws a curveball, methinks. That's a really recent version of Windows to have no support [18:58:34] Yeah :/ [18:58:40] Not in my time or sanity [18:58:40] I'm gonna guess that sets this back a few years [18:58:47] I'm guessing we still have (lots of) people using windows 7 [18:58:59] There's obviously a difference between a browser that impleemnts it for itself... [18:59:06] But tools that rely on the OS... [18:59:15] There's no TLS1.3 in IE11 or old Edge [18:59:22] You can do Win 7 through Chrome or Firefox I believe from my research [18:59:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10510080 (10Jhancock.wm) I have the two Dell Poweredge R 440 servers set aside when we are ready to rack them. they have 10G car... [19:00:20] Reedy: If you could make a version of AWB that works with 4.6.2, I will owe you a drink when I finally make it to an in person event [19:00:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10510086 (10Jhancock.wm) also forgot to mention we have one spare SFP-100G-LR4 we can test with [19:01:00] Reedy: what User Agent does AWB use too? [19:01:06] I'll have a look a little later tonight [19:01:32] "WikiFunctions ApiEdit/{0} ({1}; .NET CLR {2}" [19:01:35] for most of it [19:01:51] It may load some stuff using the default browser agent [19:02:19] But most of it should be the WikiFunctions ApiEdit prefix [19:04:32] we had wikifunctions before wikifunctions :P [19:04:44] haha [19:06:30] 280k requests from us in the last 30 days using a UA including WikiFunctions [19:15:10] https://learn.microsoft.com/en-us/dotnet/api/system.net.securityprotocoltype?view=netframework-4.6.2 doesn't have TLS 1.3 [19:16:09] https://learn.microsoft.com/en-us/dotnet/api/system.net.securityprotocoltype?view=net-9.0&viewFallbackFrom=netframework-4.7.2-pp needs 4.7.2 [19:29:32] Or 4.8? [19:29:40] https://stackoverflow.com/questions/55240173/how-to-handle-httpwebrequest-c-sharp-with-tls-1-3 [19:30:49] Yeah, 4.8 [19:30:50] So fun [19:34:24] The limited and varied unit tests we have do run though... [19:37:49] RhinosF1: https://sourceforge.net/p/autowikibrowser/code/12731/ [19:48:27] 06Traffic, 06Infrastructure-Foundations, 06SRE: Console domain and property access request - https://phabricator.wikimedia.org/T381904#10510204 (10BCornwall) a:05BCornwall→03None [19:53:58] Reedy: do you have the built executables too pls? [19:54:11] not yet [20:28:21] 06Traffic, 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 10Experimentation Lab Radar: Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#10510313 (10mforns) Will do. [22:45:06] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10510616 (10cmooney) @fgiunchedi perhaps you might know a way to do this. We now have stats like this...