[00:07:55] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:07:55] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:07:55] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:03:25] 06Traffic, 10HaproxyKafka: Upgrade HaproxyKafka to go1.27 - https://phabricator.wikimedia.org/T443640#12563939 (10Fabfur) HaproxyKafka upgraded to 0.4.0 on all cp hosts. Graphs shows a very minor regression in message processing times, but nothing else relevant on any other metric. {F115182903} We'll keep in... [09:34:25] 06Traffic, 10HaproxyKafka: Upgrade HaproxyKafka to go1.27 - https://phabricator.wikimedia.org/T443640#12564069 (10Fabfur) 05Open→03Resolved [11:30:40] 06Traffic, 07Sustainability (Incident Followup): Add option to depool DNS hosts when DNS depooling a site - https://phabricator.wikimedia.org/T440168#12564849 (10MLechvien-WMF) [12:06:07] \o Now all refreshed from the summit, I prepared https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369893 to switch the remaining LW clusters to IPIP. As usual, a review and permission to restart pybal (or someone do the latter :)) [12:09:55] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:20:52] Seems there are multiple servers in that DC with failing cmos batteries. JennH doesn't yet have a ticket but will add me as subscriber when it's created. When I'm back in I guess I'll try to knock it back into place [14:30:28] klausman: looking. [14:30:41] does now still work for the rollout or is it too late? I can take care of the PyBal stuff [14:33:24] sure, now works [14:35:55] AIUI, the cookbook isn't really needed anymore, just like with inference-staging (Unless you want me to still run it, that's fine, too) [14:36:48] sukhe: ^^^ [14:36:56] klausman: yeah I will check once, just for my own learning on it if nothing else! [14:37:37] roger [14:44:22] klausman: doing the pybal stuff now [14:45:41] :+1: [14:49:55] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:51:36] klausman: running cookbook to check [14:56:19] klausman: codfw done, checking eqiad [15:07:45] > kernel parameter(s) net.ipv4.conf.vlan1101.rp_filter have unexpected value(s). [15:08:03] don't think this is related but yeah [15:25:25] klausman: all good at your end? [15:26:00] yep! [15:26:19] nice thanks. the other error, looking into that with Cathal but not related to you (I _think_) [15:26:33] Ack. ping me as-when needed [15:33:00] klausman: all done. thanks for moving this :) [15:42:18] 06Traffic, 10Lift-Wing, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work, 06Machine-Learning-Team (Q2 FY2026-27): Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12566294 (10klausman) 05Open→03Resolved All done and deployed, thanks to Suk... [15:45:14] 06Traffic, 06Data-Engineering: Let x_kapow_estimate be available in Turnilo - https://phabricator.wikimedia.org/T445104 (10Fabfur) 03NEW [15:46:44] 06Traffic, 10Lift-Wing, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work, 06Machine-Learning-Team (Q2 FY2026-27): Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12566395 (10ssingh) >>! In T441352#12566294, @klausman wrote: > All done and dep... [17:36:45] 06Traffic, 10MediaWiki-General, 06ServiceOps, 06MediaWiki-Core-Platform-Team (Radar): A world without external cache proxies (2026) - https://phabricator.wikimedia.org/T444990#12567241 (10MLechvien-WMF) From Serviceops perspective, we're supportive of this but won't be able to dedicate capacity in short t... [18:13:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams switch orientation migration - https://phabricator.wikimedia.org/T407794#12567577 (10RobH) 05Open→03Resolved Fixed last week by dc ops onsite in esams. I've upated the netbox entries for face of the switches [18:36:25] 06Traffic, 06WMF-Legal, 13Patch-For-Review, 07Privacy: Add no-transform to Cache-Control header - https://phabricator.wikimedia.org/T218618#12567744 (10Krinkle) >>! In T218618#7734286, @Peter wrote: > Chrome Lite pages will be removed with Chrome 100 https://support.google.com/chrome/thread/151853370/s... [20:34:47] 06Traffic: Revisit the lack of gzip compression in ATS by not removing the Accept-Encoding header in requests - https://phabricator.wikimedia.org/T445155 (10ssingh) 03NEW [20:34:54] 06Traffic: Revisit the lack of gzip compression in ATS by not removing the Accept-Encoding header in requests - https://phabricator.wikimedia.org/T445155#12568154 (10ssingh) p:05Triage→03Medium [20:42:32] 06Traffic: Revisit the lack of gzip compression in ATS by not removing the Accept-Encoding header in requests - https://phabricator.wikimedia.org/T445155#12568288 (10Vgutierrez) Another angle would be compressing in ats using zlib-ng, this would be faster than doing it on varnish with their zlib fork [21:03:56] 06Traffic: Revisit the lack of gzip compression in ATS by not removing the Accept-Encoding header in requests - https://phabricator.wikimedia.org/T445155#12568388 (10Bucheon) 천안문, 사라진 목소리, 당신도 그 일부가 될 수 있습니다. Tiananmen Square, the vanished voice, you may be a part of it too. 天安门,消失的声音,你也可以成为其中的一部分。 天安門,消失的聲音,你也可... [21:16:35] FIRING: DiskSpace: Disk space doh1002:9100:/ 6.94% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=doh1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:26:26] uh [21:29:46] ran apt clean to clean the old linux images [21:31:19] yeah I'm doing some cleanup too, I was thinking to just run it fleet wide to remove the bad one [21:31:52] there are a bunch of disk alerts [21:34:22] yeah I noticed other hosts are suffering as well [21:34:32] probably a good idea [21:34:40] /var/cache is/was full [21:41:34] RESOLVED: DiskSpace: Disk space doh1002:9100:/ 6.914% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=doh1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:55:29] 14Varnish, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 5 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12568652 (10Ejegg) Thanks @Krinkle ! Do you think there is a way for this to actually work t...