[10:55:25] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 06SRE: alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019 (10Vgutierrez) 03NEW [10:55:35] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 06SRE: alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11370189 (10Vgutierrez) p:05Triage→03High [10:56:14] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11370190 (10Vgutierrez) [10:57:18] 06Traffic, 06Data-Platform-SRE, 06SRE, 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11370193 (10Vgutierrez) [10:57:41] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11370205 (10Vgutierrez) [13:58:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-d6-eqiad reboot failed, stuck in UEFI shell - https://phabricator.wikimedia.org/T409731#11370745 (10Jclark-ctr) Swapped lswtest on Tuesday with the failed switch in D6, cabled it, and handed it over to Cathal for setup. Today, re... [14:00:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-d6-eqiad reboot failed, stuck in UEFI shell - https://phabricator.wikimedia.org/T409731#11370748 (10cmooney) >>! In T409731#11370745, @Jclark-ctr wrote: > Swapped lswtest on Tuesday with the failed switch in D6, cabled it, and han... [14:04:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-d6-eqiad reboot failed, stuck in UEFI shell - https://phabricator.wikimedia.org/T409731#11370749 (10Jclark-ctr) 05Open→03Resolved a:05cmooney→03Jclark-ctr [14:20:36] 10netops, 06Infrastructure-Foundations, 06SRE: Arelion 100G transport cr1-eqiad:et-1/1/2 <-> cr1-codfw:et-1/0/2 flapping on eqiad side [Oct 2025] - https://phabricator.wikimedia.org/T407578#11370815 (10cmooney) 05Open→03Resolved So this has bounced a few times since, however it is relatively stable.... [15:10:30] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11370970 (10Tgr) > Other parties could spam plain HTTP requests with rand... [15:27:22] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047 (10cmooney) 03NEW p:05Triage→03Medium [15:32:32] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047#11371150 (10cmooney) [15:32:58] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047#11371154 (10cmooney) [15:34:10] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047#11371173 (10cmooney) [15:40:15] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047#11371233 (10Reedy) [16:24:28] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11371439 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [16:28:25] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11371504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [16:53:55] sukhe: the current bookworm deb we use for routed-capable bird was created by the NIC.cz CI pipeline, for trixie I made a package based on 2.17.2 based from forky plus the changes from the branch, will integrate it into apt and Puppet tomorrow [16:54:47] moritzm: wow, ok. thank you! [16:55:11] I guess I will move the durum hosts to trixie first in esams/magru before the hcatpcha ones, just in case [16:58:21] sounds good! bird 2.18 still isn't released, but when it's out we'll sync up both bookworm/trixie to it [16:59:14] cool thanks! [17:11:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11371762 (10RobH) Day 5 Update: * Moved all remaining ganeti hosts today * 17 hosts moved today, 108osts remain. * All remaining hosts are either k8 hosts (i... [17:12:34] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11371766 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:15:13] hello traffic friends o/ any concerns or conflicts if I deploy an ATS Lua config change around 18:00 UTC? this would just update some Lua in-place (not touching ATS itself) [17:16:00] swfrench-wmf: should be all good, thanks for checking as always! [17:16:08] * swfrench-wmf thumbs up [17:16:09] thanks! [17:18:12] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11371802 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:18:49] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11371805 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [18:09:58] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372009 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [18:17:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Netbox Cable report - incorrectly parsing Nokia power supplies - https://phabricator.wikimedia.org/T410073 (10RobH) 03NEW [18:18:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Netbox Cable report - incorrectly parsing Nokia power supplies - https://phabricator.wikimedia.org/T410073#11372087 (10RobH) [18:21:39] 06Traffic, 06Data-Platform-SRE, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11372098 (10Ahoelzl) [18:49:14] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372191 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for... [18:57:40] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [19:48:40] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [19:51:09] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372373 (10ssingh) `hcaptcha-proxy3001` worked just fine but `hcaptcha-proxy3002` does not come... [19:59:57] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11372397 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for... [20:37:01] 10Domains, 06Traffic: URL can use another script - https://phabricator.wikimedia.org/T32766#11372533 (10BCornwall) Add @CRoslof for visibility.