[00:08:24] FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:08:24] FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:45:12] 10netops, 06Infrastructure-Foundations: Restart lsw1-a2-codfw for T437984 - https://phabricator.wikimedia.org/T439134#12367681 (10ayounsi) [07:54:26] 06Traffic, 13Patch-For-Review: Reduce heap allocations on purged TCPPurger.Send() - https://phabricator.wikimedia.org/T439254#12367714 (10Vgutierrez) Original benchmark was heavily polluted by using a httptest server. By replacing it with a naive TCP server, the original benchmark looks like this: ` BenchmarkT... [07:55:04] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12367715 (10FCeratto-WMF) a:05FCeratto-WMF→03None [07:58:33] 10netops, 06Infrastructure-Foundations: Restart lsw1-a2-codfw for T437984 - https://phabricator.wikimedia.org/T439134#12367758 (10ayounsi) 05Open→03Resolved a:03ayounsi Well, CyrusOne took care of the reboot with the power outage. ` lsw1-a2-codfw> show system uptime [...] System booted: 2026-09-23... [08:00:58] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12367825 (10ayounsi) [08:01:04] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12367826 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [08:02:48] 06Traffic: Refactoring and some other work on purged - https://phabricator.wikimedia.org/T350396#12367852 (10Vgutierrez) [08:08:24] FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:09:28] 06Traffic: Reduce goroutine churn in purged - https://phabricator.wikimedia.org/T439361 (10Vgutierrez) 03NEW [08:42:20] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368068 (10dkertesz) Reverted the change that enabled persistent stats on cp7001 and cp7011 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345486 There were mo... [09:03:25] RESOLVED: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:09:57] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12368319 (10Vgutierrez) [09:14:01] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368335 (10dkertesz) We might give this another try when we upgrade to 3.4, which marks the SHM version of the stats file as no longer experimental: > MINOR: config:... [09:31:16] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12368409 (10Jelto) [10:19:02] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368616 (10Vgutierrez) 05In progress→03Stalled [10:19:33] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12368620 (10Vgutierrez) [11:15:38] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12368818 (10Vgutierrez) [13:29:38] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12369430 (10Vgutierrez) 05In progress→03Resolved [14:24:12] hi folks! If you have some time today could you review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338937 ? Lemme know if I am missing anything :) [14:45:04] 10netops, 06Infrastructure-Foundations: Junos: ae speed not reported corectly through gNMI - https://phabricator.wikimedia.org/T439202#12369774 (10ayounsi) p:05Triage→03Medium [15:15:08] 06Traffic, 10Observability-Alerting: urldownloader service should be a paging alert - https://phabricator.wikimedia.org/T435648#12370112 (10ssingh) 05Open→03Resolved a:03ssingh urldownloader is now a paging alert. Thanks to everyone for their feedback and helping improve the documentation. [15:21:00] vgutierrez: thanks for the review <3 qq - if I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338937 and then next now, does something happen on the discovery dns side? Namely, am I going to cause a DNS issue because puppet changes something on dns*? [15:25:08] that's a good question :D [15:26:47] it doesn't look like discovery services are guarded by state: service_setup [15:26:59] yeah, service_setup should be OK [15:27:33] it's when lvs_setup goes to production that the discovery records are affected [15:27:48] so discovery records will be deleted? [15:28:12] they are effectively moving from production to service_setup [15:28:21] so the pki discovery record will be removed [15:28:47] (the service status guard is inside puppet/modules/wmflib/functions/service/get_services_for.pp BTW) [15:30:24] I left an earlier comment, this part is a bit uncertain becaues typically in the reverse step, we go from state: production to state: lvs_setup that causes the discovery records to be removed, not production -> service_setup (as in this case). but even if the records are removed, wouldn't they be recreated when we go from service_setup -> lvs_setup -> production? [15:30:25] it should be fine since all clients are not using pki.discovery.wmnet [15:30:56] (I was replying to Valentin) [15:31:35] sukhe: yes.. they will be restored as soon as state goes back to production [15:43:50] perfect, so ok if I proceed? basically first two, then I verify that the discovery records go away without big issues, and then bump to lvs_setup [15:44:06] namely, is there a merciful soul that would help me in case I need it? :D [15:45:38] elukey: we are here but note that the SRE meeting starts in 15. [15:45:45] do we want to try it tomorrow perhaps? [15:46:45] :D [15:50:42] totally forgot that sukhe is a manager now [15:51:02] can't ask for some mutiny during the SRE meeting [15:51:04] all righhtttttt [15:51:08] :D [15:51:25] I'll try it tomorrow morning so Valentin will hate me [15:51:38] elukey: you can ask for mutiny anytime but it's more like that if this goes wrong, we don't want you to be here at 6 PM :P [15:51:48] need to get some quality time with him before leaving :D [15:52:15] sukhe: yes yes you are right [15:53:59] elukey: we won't see you in AMS? :( [15:54:26] sukhe: yeah my last day is Oct 30th :( [15:55:02] sad trombone [18:20:12] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12370888 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [19:03:12] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12371092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Remove... [20:05:40] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12371432 (10CDanis) So before the big ol' power outage made everyone's day a lot m... [20:35:09] 06Traffic, 06SRE: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672#12371697 (10CDobbins) @cmooney, this should be fixed now. As of 20:20 UTC today, /var/lib/prometheus/node.d/ferm-mss.prom's contents on clouddumps1002 were: ` # HELP ferm_mss_cfg MSS values for Ferm-... [21:02:36] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12371812 (10Ladsgroup) [21:03:57] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12371816 (10Ladsgroup) Thank you for the investigation Chris! <3 I updated the title to reflect the fact that thumb migration is likely not the... [21:05:52] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12371827 (10taavi) xref {T439239} [23:56:28] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#12372155 (10BCornwall) 05Stalled→03Resolved ah, makes sense. It...