[00:56:21] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12585629 (10Eevans) I used the query workbench in logstash to run... `lang=sql SELECT `@timestamp`, wiki, message FROM logstash-* WHERE `wiki` = 'enwiki' AND `chan... [08:22:30] 06Traffic, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 06Wikidata Platform Team (Sprint 11 (2026/11/04)): ATS: route WDQSv2 public /sparql endpoints through REST Gateway - https://phabricator.wikimedia.org/T443756#12586158 (10Gehel) [09:03:14] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586274 (10ops-monitoring-bot) Host dns7001* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:18:51] I rebooted dns7001 and the icinga check "NTP peers and stratum check" is not coming back up, any idea what's up? [09:26:18] nevermind, all good! [09:34:45] vgutierrez, slyngs: re T434806 - I couldn't reproduce the issue with dns7001, can I try with doh1002 (as it's a vm it might be easier) [09:34:45] T434806: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 [09:35:15] Okay [09:35:37] thx [09:38:40] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586369 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:45:15] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586373 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:56:29] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586406 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:57:53] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586408 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [10:02:22] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586416 (10ayounsi) Test is successful. # rebooted twice doh1002 as a control. Both time BFD stayed down. # Created a file `/etc/sysctl.d/99-tes... [10:02:31] yay, test successful ! https://phabricator.wikimedia.org/T434806#12586416 [10:03:51] yay for the optimistic dad :D [10:06:03] Cool. SRE has a lot of optimistic dads :-) [10:48:13] 06Traffic, 06ServiceOps, 10Wikidata, 06Wikidata Platform Team, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12586789 (10Gehel) [11:00:16] some some more tests [11:00:22] er, doing some more tests [11:00:37] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586867 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [12:24:34] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:26:10] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:03:20] 10netops, 10Ganeti, 06Infrastructure-Foundations: Invalid private v6 IP on netflow on routed Ganeti - https://phabricator.wikimedia.org/T445660 (10ayounsi) 03NEW [14:38:21] 10netops, 10Ganeti, 06Infrastructure-Foundations: Invalid private v6 IP on netflow on routed Ganeti - https://phabricator.wikimedia.org/T445660#12587895 (10ayounsi) I manually fixed netflow4003 [14:39:16] 10netops, 06Infrastructure-Foundations: Race condition setting up IPv6 address - https://phabricator.wikimedia.org/T445667 (10ayounsi) 03NEW p:05Triage→03Medium [14:59:15] 14Varnish, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 5 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12588067 (10Krinkle) a:03Krinkle @Ejegg Yes and no. As it stand today, I think we can rel... [15:01:02] 10netops, 06Infrastructure-Foundations: IPv6: use stable link-local gateways (fe80::1) - https://phabricator.wikimedia.org/T445676 (10ayounsi) 03NEW [15:05:38] 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12588126 (10cmooney) So annoying: ` Hi Cathel, I have been working with our engineering teams and was informed that this feature is not fully suppo... [15:07:30] 14Varnish, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 5 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12588129 (10Ejegg) Interesting that the cookie would work for a few days at least. I guess t... [15:08:28] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12588133 (10ayounsi) Unfortunately it's more complicated than expected... Setting up that configuration option would trigger more of the issue docume... [15:27:58] 10netops, 06Infrastructure-Foundations: Race condition setting up IPv6 address - https://phabricator.wikimedia.org/T445667#12588225 (10cmooney) > The quickest fix forward is to modify /etc/network/interfaces and replace up ip addr add with up ip addr replace which would not fail if the IP is already present.... [15:43:11] 10netops, 06Infrastructure-Foundations: IPv6: use stable link-local gateways (fe80::1) - https://phabricator.wikimedia.org/T445676#12588348 (10cmooney) > There are 2 options for a stable IP, either use the switch side interface IP, for example 2620:0:861:13d::1, which would be similar to how it's done with v4,... [16:42:57] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12588598 (10cmooney) 05Resolved→03Open p:05Medium→03Low Nokia support actually came back, and seem to have found a sequence of operations that can get the switch in... [18:48:12] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589212 (10Dzahn) [18:54:07] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team (Kanban), 13Patch-For-Review: Implement Varnish URL rewriting for internal ESI fragment endpoint - https://phabricator.wikimedia.org/T443703#12589227 (10Catrope) To elaborate on this a little bit, @Vgutierrez and I agreed th... [19:11:58] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589287 (10Dzahn) @ssingh The current way where I upload changes, got a review and then deployed myself was perfect. That being said I only do the ones that are obvious to me and are wik... [20:11:35] 06Traffic, 06Collaboration-Services, 10MediaWiki Train Development Environment: train-dev build fails to build Gerrit image: cannot fetch Git LFS objects - https://phabricator.wikimedia.org/T443677#12589518 (10Dzahn) a:03Southparkfan [20:28:09] FIRING: LVSHighRX: Excessive RX traffic on lvs6001:9100 (ens3f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:30:09] FIRING: [2x] LVSHighRX: Excessive RX traffic on lvs1017:9100 (enp59s0f0np0) - https://bit.ly/wmf-lvsrx - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:30:30] FIRING: [4x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 8 unhealthy realservers pooled on lvs6001:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [20:33:09] FIRING: [5x] LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:35:09] RESOLVED: [2x] LVSHighRX: Excessive RX traffic on lvs1017:9100 (enp59s0f0np0) - https://bit.ly/wmf-lvsrx - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:35:30] RESOLVED: [6x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 9 unhealthy realservers pooled on lvs6001:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [20:38:09] RESOLVED: [5x] LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:52:41] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589661 (10bking) `matomo-next.wikimedia.org` will soon be used, see T436003. [[ https://gerrit.wikimedia.org/r/c/operations/dns/+/1352139 | The Gerrit change that added this record ]] h... [20:58:55] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589668 (10Eevans) Looking at the most recent requests that //actually// had missing results (rows count of 0)... `lang=shell-session $ egrep "^0 " counts.txt |he... [21:08:42] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589799 (10Eevans) Following up on my original comment: >>! In T444597#12548317, @Eevans wrote: > Querying Cassandra directly: Technically the example provided i... [21:14:31] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589853 (10Dzahn) Thanks! I am going to check it off on the list. [21:15:07] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589855 (10Dzahn) [21:23:16] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589888 (10Eevans) ## TL;DR Of the 1503 logging events captured starting Nov 1, all but 225 seem to be erroneous. Meaning, the error reports no suggestions when... [22:09:41] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12590146 (10Ollie.Shotton_WMDE) Some of those `1.43.11` requests will be Wikibase Cloud instances - https://coffeebase.wikibase.cloud/wiki/S... [23:13:14] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12590372 (10colewhite) >>! In T412143#12502580, @ayounsi wrote: > We still need to upgrade more switches. Do you have an updated list, so we can compare to our upgrade plans a...