[00:02:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11338235 (10Papaul) [00:04:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11338243 (10Papaul) @cmooney thanks for the feedback we can clarify this tomorrow during the meeting and have all ready and run it by @ayounsi when he is back. [00:07:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11338254 (10Papaul) [00:10:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11338258 (10Papaul) [00:58:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [01:03:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:06:04] 06Traffic, 06SRE, 05FY2025-26 WE3.3 Engaging core audiences, 06Reader Experience Team (REx Sprint 8 [Q2 Oct 21-Nov 3]): [Reading Lists] Monitor potential performance impact of Reading Lists for Web - https://phabricator.wikimedia.org/T397526#11338576 (10Jdrewniak) Thanks for those graphs @SToyofuku-WMF , l... [10:03:35] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11339074 (10Aklapper) @Shisma: Hi, can you please answer on this ticket? Otherwise it will be declined. Thanks. [10:03:46] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11339075 (10Aklapper) a:03Shisma [12:10:51] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Distinguish request classes based on user-agent declaration - https://phabricator.wikimedia.org/T408060#11339513 (10Fabfur) 05Open→03Resolved a:03Fabfur haproxy configuration deployed everywhere [12:23:49] 06Traffic: Implement HEAD→GET caching logic implemented against WME globally - https://phabricator.wikimedia.org/T382276#11339590 (10LSobanski) Removing the #wikimedia-incident-actionable tag. If this is still a solution to a possible incident root cause please add the tag back and consider raising the task pri... [12:24:19] 06Traffic: Add Wikimedia Enterprise IPs as an ipblock in requestctl - https://phabricator.wikimedia.org/T382275#11339593 (10LSobanski) Removing the #wikimedia-incident-actionable tag. If this is still a solution to a possible incident root cause please add the tag back and consider raising the task priority. [13:37:28] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178 (10cmooney) 03NEW p:05Triage→03Medium [13:39:40] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11340040 (10cmooney) [15:31:34] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice-archive: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11340484 (10CKoerner_WMF) Could I request assistance in verifying that we've properly addressed the user-agent policy requirements for Diff's o... [15:38:28] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11340506 (10akosiaris) >>! In T352956#11247042, @ssingh wrote: > Hi @akosiaris: Following up on this after a dis... [16:33:29] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11340850 (10BCornwall) I'm going to first verify that this was intentional before merging. Thanks for bringing thi... [17:06:14] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11341062 (10Dzahn) We might need some coordination when wikimediafoundation.org makes changes on their end. @SCamp... [17:37:00] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11341304 (10SCampos-WMF) Hi @Dzahn, thanks for the ping. This change was expected, we created a redirect for "http... [17:45:03] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11341325 (10Aklapper) How / where to see such changes / find out about them? https://github.com/wikimedia/wikimedi... [17:53:17] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11341357 (10BCornwall) Now that there's a redirect I'm not sure whether we should keep or kill the ncredir redirec... [18:06:54] 06Traffic, 10wikimediafoundation.org, 13Patch-For-Review: Some domains point to Chinese version of Wikimedia Foundation website which no longer exists - https://phabricator.wikimedia.org/T407579#11341448 (10BCornwall) 05Open→03In progress p:05Triage→03Low [18:50:16] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11341689 (10VRiley-WMF) Spoke to @cmooney about this ticket. This no longer has to be move... [18:51:24] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11341696 (10VRiley-WMF) Spoke to @cmooney about this ticket. This no longer has to be move... [18:58:04] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11341710 (10cmooney) >>! In T405628#11341689, @VRiley-WMF wrote: > Spoke to @cmooney about... [19:27:48] 06Traffic, 13Patch-For-Review: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11341777 (10BCornwall) 05In progress→03Resolved [21:03:35] 06Traffic, 10DNS: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11342081 (10SCampos-WMF) Hi @BCornwall thank you for this. I have one last update request, apologies for the back and forth. I just heard from Fundraising t... [21:05:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11342084 (10cmooney) >>! In T405609#11341696, @VRiley-WMF wrote: > Spoke to @cmooney about... [21:13:16] 10netops, 06Infrastructure-Foundations, 06SRE: Rancid network backups not being synced to git properly - https://phabricator.wikimedia.org/T409217 (10cmooney) 03NEW p:05Triage→03Medium [21:20:50] 10netops, 06Infrastructure-Foundations, 06SRE: Rancid network backups not being synced to git properly - https://phabricator.wikimedia.org/T409217#11342153 (10Dzahn) I would expect the cause is that someone committed as root: ` root@netmon1003:/var/lib/rancid/core/.git/objects# find . -user root ./f2 ./f2/... [21:31:16] hello traffic friends - FYI, in a couple of minutes, I'll be running a fairly narrowly scoped test of a new requestctl feature on cp2041. this is enabled only on that host, so this will *not* involve messing with puppet across the whole of A:cp. [21:42:34] cool! [22:17:05] 10netops, 06Infrastructure-Foundations, 06SRE: Rancid network backups not being synced to git properly - https://phabricator.wikimedia.org/T409217#11342444 (10cmooney) Thanks @Dzahn appreciate it! Yep that's what I thought, I will give your suggestion a try in the morning and see does it resolve the problem. [22:42:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [22:47:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [22:54:34] (cross-posting) all done messing with cp2041 [23:51:43] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Introduce known-client identity objects and integrate with requestctl - https://phabricator.wikimedia.org/T403220#11342816 (10Scott_French) Alright, progress! Aside from the oversight addressed in https://gerrit.wikimedia.org/r/1201811 that initially prevented c... [23:56:51] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11342826 (10Dzahn) haproxy configured as TCP-proxy for gerrit-ssh has been deployed on all 14 VMs, across POPs. We can now con... [23:57:30] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11342827 (10Dzahn)