[02:04:27] Could use some review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1355384/ basically takes the current kafka-jumbo topics and encodes their state in hiera. (topics w/ only defaults are excluded) [08:45:28] ryankemper: maybe that's something elukey or brouberol can help review? the most recent change regarding partitions on the exporter side is https://github.com/wikimedia/operations-puppet/commit/769997dc4dd5ce717a5c81e54bb688c61702da3d which was duplicated in https://github.com/wikimedia/operations-puppet/blob/a6c8f01a84480dd6db5cdcc461b197de4be0d4a3/modules/pmacct/templates/sfacctd.conf.erb#L33 [08:46:33] I think data engineering has more knowledge on the topic than I/F (especially starting next week :( ) [08:49:03] we are having some issues with cookbooks based on `SRELBBatchRunnerBase`, sometimes after what looks like a successful operation the server never gets repooled [08:50:07] cp2049 never got repooled last night, even if the cookbook logged it: https://www.irccloud.com/pastebin/PGzgFWsp/ [08:51:10] in the extended log I didn't find any relevant error for the etcd.client [08:52:59] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12483112 (10ayounsi) Sent https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1359754 for the repool. @wiki_willy issue lasted 2h50min, do you think it's wo... [09:01:02] ryankemper XioNoX: yeah this is small enough that I can easily review, no worries, I 'll take care of it [09:01:29] <3 [09:45:49] +1ed after checking the kafka live state [09:49:59] thx [09:51:36] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12483296 (10ayounsi) a:05ayounsi→03wiki_willy Circuit is repooled, feel free to close the task at your discretion. [10:51:27] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691 (10ayounsi) 03NEW p:05Triage→03Medium [10:55:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483604 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cd2fa008-95d5-4a9d-8... [10:58:23] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691#12483625 (10ayounsi) [11:19:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483741 (10cmooney) [11:51:07] FIRING: [2x] ProbeDown: Service rpki1001:3323 has failed probes (tcp_rpkivalidator_rtr_ip4) - https://wikitech.wikimedia.org/wiki/RPKI#RPKI_to_router_port - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:01:12] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:03:17] RESOLVED: [2x] ProbeDown: Service rpki1001:3323 has failed probes (tcp_rpkivalidator_rtr_ip4) - https://wikitech.wikimedia.org/wiki/RPKI#RPKI_to_router_port - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:11:57] FIRING: [3x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:14:25] FIRING: [3x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:25] RESOLVED: [2x] SystemdUnitFailed: netbox_report_librenms_run.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:41:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484047 (10cmooney) [12:44:41] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705 (10LSobanski) 03NEW [12:53:37] vgutierrez: was that as part of the run in SAL from bre.tt at 19:46-20:41 running sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade ? [12:57:22] if that's the one, is it possible that cp2049 was already depooled? I don't see the log that changed it to depooled, and hence it was not restored to pooled. [12:58:37] from https://grafana.wikimedia.org/goto/sblznb?orgId=default it seems it was not getting traffic from the morning, and from SAL there was a roll reboot going on [12:59:12] you can see the confctl changes with: grep 'Updating conftool' roll-upgrade-ats-extended.log [13:00:56] related SAL with what looks like some manual intervention for cp2049 in https://sal.toolforge.org/production?p=0&q=cp2049&d= [13:36:03] volans: previous cookbook depooled it, got canceled so the next cookbook execution never repooled it [13:47:43] vgutierrez: ok that explains it right? [14:40:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484789 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bb0696e8-7843-4... [14:40:46] volans: yes, sorry I wasn't clear, L8 on our side :) [14:40:55] :D [14:40:59] no prob [14:41:34] arguably it could log it that the confctl object was already in the desired state and no changes were made [14:48:17] elukey: BTW.. I reimaged lvs5005 without issues this morning and now I have the same issues with lvs5004.. it failed to fetch the complete initrd... what's the expectations in terms of BW to be able to fetch it without the client bailing out? [14:49:07] initrd.gz is fairly big nowadays (513MB) [14:50:09] wow [14:51:08] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12484844 (10Joe) A question to answer is: why was this done before the switchover of all services to codfw that would've happened a few hours later? Or - be... [14:52:07] apt.wm.o is running with the default nginx timeouts apparently with just a explicitly set `keepalive_timeout 65` [14:52:56] but nginx doesn't have a timeout for a full HTTP request, so it should be OK on that side [15:18:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484989 (10cmooney) [15:21:31] so.. this time the cookbook detected the debian installer on attempt 118 of 240.. on the previous one it detected the old system on attempt 39 of 240.. so it looks like it isn't a timeout but the connection being closed for some reason and not retrying to fetch it [15:29:34] vgutierrez: yeah I think the iPXE logic is not going to retry etc.. [15:30:14] I'd suggest, if possible, to start using UEFI sooner rather than later, the BMC support for it is way better [15:30:21] All HTTP based etc.. [15:30:44] have you tried the http parameter as well with lvs5004, or was it done via tftp? [15:30:44] elukey: as in upgrading idrac/bios on old systems? [15:30:54] I've used the http paremeter yes [15:31:01] *parameter evefn [15:31:05] urgh.. nevermind [15:31:33] vgutierrez: no no it should be sufficient to run provision to flip UEFI settings on, and then amend the partman recipe to take into account the uefi partition [15:31:37] and reimage [16:23:55] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12485418 (10cmooney) Thanks for the task @LSobanski To confirm what happened here I tried to be "smart" and deviated from my steps as set out in the task l... [16:39:48] FIRING: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:55:33] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12485635 (10Lizimedia) to summarize our discussion from the most recent SRE Tooling and Automation Office Hours: Whenever a change to the cookbooks repo... [17:19:08] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12485760 (10cmooney) >>! In T442705#12484844, @Joe wrote: > A question to answer is: why was this done before the switchover of all services to codfw that wo... [18:24:48] RESOLVED: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:47:33] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12486195 (10cmooney) [18:47:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486194 (10cmooney) [19:41:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486364 (10cmooney) [19:56:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486454 (10cmooney) 05Open→03Resolved All done with this one. Unfortunately interr... [23:35:30] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12487322 (10wiki_willy) Thanks, I'll go ahead and request the credit. Did they provide you a root cause for the issue?