[07:49:41] 06Traffic, 06MW-Interfaces-Team, 10ServiceOps-SharedInfra, 06ServiceOps new (Next quarter): map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#12008514 (10Clement_Goubert) [07:57:29] 10Wikimedia-Apache-configuration, 06ServiceOps new, 10ServiceOps-good-first-task, 10ServiceOps-Mediawiki, 06MediaWiki-Platform-Team (Radar): Serve mediawiki keys.txt with UTF-8 charset - https://phabricator.wikimedia.org/T428772#12008536 (10Clement_Goubert) p:05Triage→03High [08:52:09] 10Wikimedia-Apache-configuration, 06ServiceOps new, 10ServiceOps-good-first-task, 10ServiceOps-Mediawiki, and 2 others: Serve mediawiki keys.txt with UTF-8 charset - https://phabricator.wikimedia.org/T428772#12008740 (10Clement_Goubert) [09:48:29] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12008995 (10Gehel) [09:50:08] Hello traffic friends! We have an ask to do a daily import of Google Search Console data - T427457. The estimate is about 300GB daily. Before moving forward, we'd like you to review this and let us know what constraints we have. [09:50:09] T427457: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457 [10:41:37] 06Traffic, 10Incident Tooling: Proof of Concept: SquareOne CDN Dashboards - https://phabricator.wikimedia.org/T414665#12009164 (10jijiki) [11:11:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12009270 (10cmooney) [11:12:06] 06Traffic, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12009273 (10ops-monitoring-bot) Draining ganeti5006.eqsin.wmnet of running VMs [11:12:37] 06Traffic, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12009289 (10MoritzMuehlenhoff) [11:13:00] 06Traffic, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12009290 (10ops-monitoring-bot) Draining ganeti5006.eqsin.wmnet of running VMs [11:15:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12009300 (10cmooney) @papaul if you get a few minutes to double-check the above let me know. And specifically on the phys... [11:16:47] 10Wikimedia-Apache-configuration, 06SRE: Move kr.wikimedia destination to [[m:Wikimedia Korea]] - https://phabricator.wikimedia.org/T428327#12009320 (10revi) 05Open→03Resolved Was merged, the metawiki-side switchover happened, nothing left here. Closing. [11:58:51] 10netops, 10homer, 06Infrastructure-Foundations, 06SRE: Homer should abort on filter rules applied on non-existent or disabled interfaces - https://phabricator.wikimedia.org/T428886#12009524 (10cmooney) p:05Triage→03Medium Thanks @taavi Yes we can do some validation in Homer to avoid this I think, I'... [12:08:27] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 13Patch-For-Review: codfw: rack A5 maintenance - https://phabricator.wikimedia.org/T428020#12009567 (10MoritzMuehlenhoff) @ayounsi For puppetserver2002 will need to be merged before the maintenance starts: https://gerrit.wikimedia.org/r/c/operations... [12:20:30] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12009615 (10ayounsi) @Antoine_Quhen Our network can 100% handle that kind of load, but we have some question... [12:26:22] 06Traffic, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12009663 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti5006.eqsin.wmnet with OS bookworm [13:24:08] 06Traffic, 06collaboration-services, 10GitLab, 06Release-Engineering-Team (Radar): gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903 (10ABran-WMF) 03NEW [13:25:01] 06Traffic, 06collaboration-services, 10GitLab, 06Release-Engineering-Team (Radar): gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12009971 (10ABran-WMF) 05Open→03In progress p:05Triage→03High [13:32:03] 06Traffic, 06collaboration-services, 10GitLab, 06Release-Engineering-Team (Radar): gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12010021 (10ABran-WMF) [13:45:32] 10netops, 10homer, 06Infrastructure-Foundations, 06SRE: Homer should abort on filter rules applied on non-existent or disabled interfaces - https://phabricator.wikimedia.org/T428886#12010069 (10ayounsi) Quick update after chatting about that with Cathal. For context the current implementation looks like:... [13:46:35] 06Traffic, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12010071 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti5006.eqsin.wmnet with OS bookworm completed: - gan... [13:50:59] 06Traffic, 10DNS, 06SRE: 10.67.28.73 reverse DNS showing 2(SERVFAIL) - https://phabricator.wikimedia.org/T428573#12010077 (10CDanis) >>! In T428573#12001514, @cmooney wrote: > It doesn't seem to have a service endpoint registered though, which I think is needed before CoreDNS will publish any records for it:... [14:01:18] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: check if we need to filter irb interfaces for DHCP relay / IPv6 RA - https://phabricator.wikimedia.org/T428908 (10cmooney) 03NEW p:05Triage→03Low [14:38:09] 06Traffic, 06Infrastructure-Foundations: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12010369 (10MoritzMuehlenhoff) [14:38:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12010370 (10ayounsi) Minor, but it might also be a good opportunity to inspect the air filters: https://www.juniper.net/do... [14:57:29] 10netops, 06Infrastructure-Foundations: Add (some) collection for Nokia SR-Linux components - https://phabricator.wikimedia.org/T428685#12010472 (10cmooney) We rolled back the patch here as it triggered the bug we previously seen collecting the components path from Nokia. Restricting the particular components... [15:00:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12010478 (10cmooney) >>! In T426343#12010370, @ayounsi wrote: > Minor, but it might also be a good opportunity to inspect... [15:11:55] gehel: ack on the question, we'll chime in to the ticket. Thanks! [15:14:25] FIRING: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp3080:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:14:41] uh? [15:14:49] are we working on cp3080? [15:16:41] no [15:17:16] wonder why varnish wants to visit the hospital. let's check [15:17:23] > Service varnish-frontend-hospital not present or not running [15:17:34] that's all the journal has [15:17:58] fun times [15:18:49] restarting wmf_auto_restart_varnish-frontend-hospital.service and it's happy again... [15:18:54] yeah weird [15:18:57] ok let's see if it happens again [15:18:58] thanks [15:24:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp3080:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:25:22] brett: https://w.wiki/Q$Cy [15:25:29] looks a bit weird for that time don't you think? [15:30:22] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12010744 (10BCornwall) a:05BCornwall→03MShilova_WMF Hi, @MShilova_WMF! @SLyngshede-WMF and I have a patch ready for deployment - this deployment/enforcement will patiently wait until your signal. [15:43:01] sukhe: you mean the dip? [15:43:42] yeah [15:44:54] Jun 11 15:09:44 cp3080 purged[2781426]: 2026/06/11 15:09:44 Error connecting to unix:/run/varnish-privileged.socket: dial unix /run/varnish-privileged.socket: connect: connection refused. Reconnecting in 4096 millisecond [15:45:14] Jun 11 15:09:40 cp3080 varnishd[1932]: Child (2767) said Child dies [15:45:22] ouch [15:45:34] so yeah we need to look what happened here [15:45:59] maybe blblack's restart? [15:46:12] Jun 11 15:08:53 cp3080 varnishd[1932]: Manager got SIGTERM [15:46:14] Jun 11 15:08:53 cp3080 systemd[1]: Stopping varnish-frontend.service - varnish-frontend (Varnish HTTP Accelerator)... [15:46:17] yeah matches that [15:46:18] :) [15:46:25] is blblack's cookbook running on esams? [15:46:54] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12010832 (10Gehel) >>! In T427457#12009615, @ayounsi wrote: > * What's the timeline for the project ? Ideally... [15:47:16] * brett checks [15:48:35] hm, one screen session is on eqsin, not sure where the second one is (it's only reporting one in 'screen -ls') [15:49:08] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12010852 (10xcollazo) CC @BTullis, for visibility. [15:50:51] sukhe: I do believe it was bblack's cookbook run though as apt's history log has it installing libvmod-wmfuniq [15:50:58] ok thanks for checking [15:51:05] that should be it then [15:51:13] yeah, it's servicing 3081 now [16:23:21] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12011063 (10Ejegg) We definitely want to do this as soon as it's convenient for the core team. It'll help cut down on tr... [16:25:51] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12011077 (10AKanji-WMF) Let's aim for Q1 [16:28:56] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12011102 (10ssingh) This will require support from Traffic in some capacity, so please let us know and we can prioritize... [17:09:03] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12011346 (10BCornwall) @Gehel Thanks for looping in traffic. If I'm reading this correctly, the Hadoop cluster... [17:32:39] 10Wikimedia-Apache-configuration, 06ServiceOps new, 10ServiceOps-good-first-task, 10ServiceOps-Mediawiki, and 2 others: Serve mediawiki keys.txt with UTF-8 charset - https://phabricator.wikimedia.org/T428772#12011459 (10Blake) Looks like my patch didn't work - the charset parameter wasn't added to the Cont... [17:34:45] 06Traffic, 06Infrastructure-Foundations: eqsin: re-image rack 604 servers on new vlan - https://phabricator.wikimedia.org/T428229#12011469 (10BCornwall) [19:04:14] 10Wikimedia-Apache-configuration, 06ServiceOps new, 10ServiceOps-good-first-task, 10ServiceOps-Mediawiki, and 2 others: Serve mediawiki keys.txt with UTF-8 charset - https://phabricator.wikimedia.org/T428772#12011811 (10matmarex) When making changes to keys.txt the other day, we had to manually purge the c... [19:21:28] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12011841 (10MShilova_WMF) @BCornwall , sounds good. Thank you! I'll update the ticket once we are ready to proceed with the deployment. [19:29:46] 10Wikimedia-Apache-configuration, 06ServiceOps new, 10ServiceOps-good-first-task, 10ServiceOps-Mediawiki, and 2 others: Serve mediawiki keys.txt with UTF-8 charset - https://phabricator.wikimedia.org/T428772#12011885 (10RLazarus) >>! In T428772#12011811, @matmarex wrote: > When making changes to keys.txt t... [19:55:40] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12011986 (10CDanis) The MR looks good to me! I'm happy to help with the haproxy patch portion of possibility 1, which is my prefe... [20:47:36] 06Traffic, 05Bot detection and mitigation (WE4.10 hCaptcha), 07Documentation, 06Product Safety and Integrity (Sprint Iris (May 25 - Jun 12)): hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#12012223 (10Dreamy_Jazz) a:05Dreamy_Jazz→03None Unassigning from myself based on... [21:44:38] 06Traffic, 06SRE, 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [puppet] Remove expired and unused certs from modules/profile/files/ssl/ and modules/base/files/ca - https://phabricator.wikimedia.org/T354295#12012402 (10BCornwall)