[00:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:51:47] hello folks, I'm doing some cleanup for the `wikimedia_clusters` hiera key and noticed that the pybal profile has an if clause checking if the current site has any mediawiki appservers (which haven't existed in reality for a while as we do mw-on-k8s now). https://gerrit.wikimedia.org/r/c/operations/puppet/+/1370075 removes the check, should be a [11:51:47] noop but would appreciate your +1 to be sure [11:54:08] taavi: I suspect that we can drop runcommand [11:55:02] yeah.. we don't have in place any RunCommand monitor [11:55:15] and TBH I don't think we had one in the last ~9 years [11:55:42] that matches with what cumin sees: https://phabricator.wikimedia.org/P97468 [11:55:53] want me to send a patch to drop the directory instead? [11:56:14] yes please [11:59:53] vgutierrez: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1370473 [12:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:07:59] taavi: thx <3 [12:08:42] vgutierrez: do you think that's safe enough to just merge and let it roll out or should I do the dance of disabling puppet, merging, rolling out on one and testing a pybal restart, etc? [12:08:58] safe enough IMHO [12:09:01] cool [12:25:04] I'm rolling out a NOOP change to all the PyBal BGP group: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1361729 - https://phabricator.wikimedia.org/T443003 [12:27:05] cool :D [12:29:06] confirmed it didn't cause any bgp session bounce on asw1-b12-drmrs, rolling it out everywhere [12:46:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp7011:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [12:46:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp7011 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=magru&var-instance=cp7011 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [13:07:45] 10netops, 06Infrastructure-Foundations: Investigate enabling multipath for Pybal BGP group - https://phabricator.wikimedia.org/T443003#12570815 (10ayounsi) 05Open→03Resolved a:03ayounsi Rolled out on all the Juniper devices, and Nokia already have it configured. All done. [13:25:39] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12570884 (10ayounsi) While I think about it, the workflow will also need to run the provision cookbook to update the server's PXE interface,... [14:37:43] hello puppet has a script `utils/vcl_ec2_nets.py` for which I would need the shebang to be updated (from `python` to `python3` ). That'll let me drop Python 2 support from CI [14:37:44] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364492 [14:38:21] though maybe that script is no more of any use, it seems to fetch IP ranges from AWS to generate some VCL rule, and my guess is it is no more needed [14:39:22] in which case the script should be deleted [14:42:09] doesn't looks like we have an acl called "aws_nets" AFAICT, but maybe I'm not looking in the right places [14:46:23] that script could be removed AFAIK [15:00:43] hashar: could you git rm that script instead please? [15:00:54] sure! [15:04:45] 06Traffic, 06MediaWiki-Core-Platform-Team (Radar): Measure client clock skew (2026) - https://phabricator.wikimedia.org/T445220 (10Krinkle) 03NEW [15:06:40] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12571359 (10ayounsi) Not sure if that will help much, but I opened JTAC case 2026-1117-230670 about a sensord core dump on cr1-eqiad. Maybe it's r... [15:07:03] vgutierrez: I have amended my change to delete the script https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364492 [15:07:03] :) [15:07:51] can you +2/merge it? I don't have puppet-merge rights [15:08:48] 06Traffic, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add `x_kapow_estimate` to `webrequest` - https://phabricator.wikimedia.org/T445221 (10JAllemandou) 03NEW [15:14:47] hashar: done [15:14:53] awesome [15:14:55] thank you! [15:31:03] fabfur: you seem to have an un-puppet-merge change... Something about benthos and kapow. Can I safely merge it? [15:31:20] oops sorry, completely passed [15:31:26] thanks, can be merged! [15:31:35] done [15:31:38] thanks! [15:51:18] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12571629 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos [15:52:07] 06Traffic, 06MediaWiki-Core-Platform-Team (Radar): Measure client clock skew (2026) - https://phabricator.wikimedia.org/T445220#12571649 (10Krinkle) I suggest the following instrument, delivered via the `ext.wikimediaEvents` module, for upto 1 week. `lang=js async function kfReportClientClockSkew() { if (!mw... [16:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:25] 06Traffic, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add `x_kapow_estimate` to `webrequest` - https://phabricator.wikimedia.org/T445221#12571742 (10JAllemandou) [16:13:15] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12571810 (10cmooney) 05Open→03Resolved I'm gonna close this one given the issue has not re-occured, but we need to be aware there is likely some real edge-case bug... [16:20:21] 06Traffic: Revisit the lack of gzip compression in ATS by not removing the Accept-Encoding header in requests - https://phabricator.wikimedia.org/T445155#12571840 (10Vgutierrez) Expanding on my previous comment: the CDN currently receives all content uncompressed, and it is compressed by Varnish using its intern... [17:55:06] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: Support x-kapow-score header in HaproxyKafka - https://phabricator.wikimedia.org/T443632#12572518 (10Fabfur) 05Open→03Resolved [17:56:29] 06Traffic, 06Data-Engineering: Let kapow_estimate be available in Turnilo - https://phabricator.wikimedia.org/T445104#12572542 (10Fabfur) [20:06:10] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed