[05:35:41] 06Traffic, 06Data-Platform-SRE, 06Wikidata Platform Team (Sprint 11 (2026/11/04)): ATS: route WDQSv2 public /sparql endpoints through REST Gateway - https://phabricator.wikimedia.org/T443756 (10gmodena) 03NEW [05:36:37] 06Traffic, 06Data-Platform-SRE, 06Wikidata Platform Team (Sprint 11 (2026/11/04)): ATS: route WDQSv2 public /sparql endpoints through REST Gateway - https://phabricator.wikimedia.org/T443756#12520953 (10gmodena) [09:25:18] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691#12521384 (10ayounsi) [10:09:01] 06Traffic: the configuration file for check_rp_filter_disabled can be too large for icinga - https://phabricator.wikimedia.org/T443771 (10dkertesz) 03NEW [10:16:37] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521571 (10cmooney) [10:16:40] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12521570 (10cmooney) [10:17:11] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521573 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=351d61a3-389f-44b4-9fe8-5fea6f1228f9) set by cmooney@cumin1004 for 1:00:00 on 1 host(s) and the... [10:40:30] 06Traffic, 06Commons, 06MediaWiki-Media-Platform-Team: SVG thumbnail downloads as WebP instead of PNG in Chrome browsers - https://phabricator.wikimedia.org/T443394#12521640 (10JTweed-WMF) p:05Triage→03Low [11:17:31] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance: ESI: Enforce public blocking of both /esi and internal fragment rotes - https://phabricator.wikimedia.org/T443704#12521759 (10Vgutierrez) 05Open→03In progress a:03Vgutierrez [11:17:53] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance: ESI: Enforce public blocking of both /esi and internal fragment routes - https://phabricator.wikimedia.org/T443704#12521762 (10Vgutierrez) [11:22:43] o/ is there a process around adding a new component that uses the urldownloaders? I'm working with a dev team that will need to make some external requests, relatively low impact and just want to check in on it [11:54:23] 06Traffic, 13Patch-For-Review: the configuration file for check_rp_filter_disabled can be too large for icinga - https://phabricator.wikimedia.org/T443771#12521835 (10dkertesz) we're taking the chance to move this alert to Prometheus, which was meant to happen anyway eventually; see T407330 [11:54:39] 06Traffic, 13Patch-For-Review: the configuration file for check_rp_filter_disabled can be too large for icinga - https://phabricator.wikimedia.org/T443771#12521837 (10dkertesz) 05Open→03In progress a:03dkertesz [11:57:09] hnowlan: not that I'm aware but it should be fine [11:59:35] vgutierrez: cool, thanks! [12:08:53] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 13Patch-For-Review: ESI: Enforce public blocking of both /esi and internal fragment routes - https://phabricator.wikimedia.org/T443704#12521867 (10Vgutierrez) 05In progress→03Resolved ` vgutierrez@cp3066:~$ curl https://en.wikipedia.org/esi/foo -s -... [14:30:48] 10netops, 06Infrastructure-Foundations, 07Incident Severity 2, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12522432 (10MLechvien-WMF) [14:57:44] \o In the context of https://phabricator.wikimedia.org/T441352 (IPIP for inference-staging) I'd like to have a review of merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364397 and the permission to then run the sre.loadbalancer.migrate-service-ipip cookbook. TIA! [14:59:27] 06Traffic, 10Lift-Wing, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work, and 2 others: Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12522616 (10klausman) a:03klausman [15:03:18] thanks, v_gutierrez! [15:03:27] klausman: LGTM, migrate-service-ipip is no longer needed for you IIRC [15:03:35] your realservers already handle IPIP traffic [15:03:52] I wasn't quite sure. So just merging the patch is enough? [15:04:06] so as soon as you merge the change pybal needs to be restarted on lvs2013 and lvs2014 [15:04:13] (and puppet runs there ofc) [15:04:25] ack. [15:04:37] I can handle the restarts for you if you're merging it right now [15:04:58] otherwise you got instructions on wikitech and/or my teammates can help you :D [15:05:45] I'll merge in a second [15:09:23] merged and ran puppet-agent on lvs2013 and lvs2014. pybal restarts always scare me, so thanks for taking care of that :) [15:09:41] FIRING: AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:10:11] vgutierrez: only one left? :) https://phabricator.wikimedia.org/T441429 [15:11:39] klausman: np :D [15:12:14] inference-prod still needs doing, but that'll I leave for monday :) [15:15:37] klausman: done :D [15:15:57] Gracias! [15:19:32] thanks for taking care of it klausman! [15:19:39] XioNoX: that's on me :) [15:19:41] FIRING: [3x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:28:11] ^ this is related to https://gerrit.wikimedia.org/r/c/operations/alerts/+/1364303 but I still don't know why [15:29:41] FIRING: [4x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:34:41] FIRING: [5x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:34:41] FIRING: [5x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:39:41] FIRING: [5x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:49:12] hi I am not sure who maintains pybal nowadays, but I am assuming that is Traffic [16:49:41] FIRING: [5x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:49:45] hashar: that's us for sure :) [16:49:47] what's up? [16:49:54] there is an Icinga check that is based on Python 2 and I proposed a change to port it to python3 (lvs servers have python 3.9) [16:49:57] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364471 [16:50:18] the reason for the change is that I am looking at removing Python2.7 from the CI image used to run tests on operations/puppet :b [16:50:23] yeah thanks, we can look [16:50:31] Pybal itself is based on Python2 but that's not related to this change :) [16:50:49] it is not super urgent critical and I don't have access to the lvs server to test it :) [16:51:05] no worries, thanks for the patch [16:51:08] we will take care of the roll out! [16:51:14] awesome thank you :] [16:51:31] there are a few more scripts that still assume python 2, but that will be for another day :) [16:52:06] hmm [16:52:19] yes, we do have python3 on the LVSes ok [16:52:25] the core ones on bullseye [16:59:41] RESOLVED: [3x] AlertLintProblem: Linting problems found for LVSRpFilterEnabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:00:04] and another which sounds like Varnish is utils/vcl_ec2_nets.py which has a shebang of `python` [17:00:22] CI lints it as Python2 currently and will later on lint it as python3 [17:00:39] so the shebang should be changed python > python3 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364492 ;) [17:01:03] I think that is all I have for Traffic ;) [17:01:13] thanks :) [17:10:14] 06Traffic, 10HaproxyKafka: Upgrade HaproxyKafka to go1.27 - https://phabricator.wikimedia.org/T443640#12523725 (10Fabfur) HaproxyKafka 0.4.0 compiled for golang 1.27 has been deployed on cp7002 to test for eventual regressions or performance issues. This can be upgraded everywhere in the next days. This versio... [17:32:00] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523829 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=37a3a0c5-db75-4a5f-b3af-203b15319a02) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:32:35] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523837 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a759d84c-b17a-40ad-9d3b-e8171fe3782b) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:33:17] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523840 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=83c9e035-90af-41e4-bc00-e2734e08a539) set by cmooney@cumin1004 for 5:00:00 on 2 host(s) and the... [17:35:25] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523870 (10cmooney) FWIW I did some more testing today, starting with no network-instances, irb sub-interface etc configured, and built the config up bit by bit. See P973... [18:00:05] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523955 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=866cbc1d-696d-4894-9d5d-2aa4aa1c50da) set by cmooney@cumin1004 for 3 days, 0:00:00 on 2 host(s)... [18:25:44] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team: Implement Varnish URL Rewriting for Internal ESI Fragment Endpoints - https://phabricator.wikimedia.org/T443703#12524056 (10Catrope) Yes I'm good with those rules. I had originally proposed using a path component for the user... [19:59:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3067&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:00:15] hmm [20:04:43] FIRING: [3x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:05:41] checks out with the rps [20:09:43] FIRING: [3x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:14:43] RESOLVED: [3x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:31:48] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524464 (10cmooney) In an effort to try and work out what version things still worked in I downgraded lsw1-b3-eqiad to 26.7.1, same issue. I then went to 26.3.3 - same th... [20:38:41] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524495 (10cmooney) p:05High→03Medium [20:39:27] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524496 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8099810e-6408-4be2-82b4-5240b5d6c531) set by cmooney@cum... [21:20:06] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524731 (10ops-monitoring-bot) Host lvs1019.eqiad.wmnet rebooted by cmooney@cumin1004 with reason: reconfigure vlan ints [23:46:18] hi traffic! anyone nearby to look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364993? [23:47:47] I don't think I've ever tried to edit `sites` in-place like this -- it could never have worked as-is, and the probes have probably been failing ever since it was added. but not sure what this will need to deploy, along the lines of a pybal restart