[00:15:25] FIRING: [2x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:25:25] FIRING: [2x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:14:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10612190 (10Papaul) @Jclark-ctr @VRiley-WMF the 2 switches are received in coupa but are missing in netbox. if there are not ready to be racked yet, can... [01:20:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10612192 (10Jclark-ctr) @VRiley-WMF I have not seen these in the data center yet but you updated ticket Jan 10 2025 almost 2 months ago? Receiving ti... [04:20:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:42:21] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC policy - https://phabricator.wikimedia.org/T378285#10612363 (10revi) And the result seems fine! `lang=eml, lines=7 Authentication-Results: phl-mx-03.messagingengine.com; dkim=pass (1024-bit rsa key... [14:00:55] 10netops, 06Infrastructure-Foundations, 10ops-magru: Jan 2025 - Magru core router connectivity blips - https://phabricator.wikimedia.org/T384774#10613224 (10cmooney) p:05High→03Medium Everything still seems table. Juniper also provided this link to their KB article on it https://supportportal.juniper.n... [14:59:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10613477 (10cmooney) Thanks guys. Please ping me when these are in Netbox and I will add the links, IPs, vlans etc. and begin the process of commissioni... [15:36:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10613663 (10cmooney) Lastly please call these new switches //lsw1-e8-eqiad// and //lsw1-f8-eqiad// in Netbox. We'll need to either have deleted the Dell... [15:45:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10613705 (10VRiley-WMF) Will be adding these into netbox shortly [15:59:20] per the discussion on https://phabricator.wikimedia.org/T385995 the gitpuppet group is not currently being created on the puppetservers, because of an oversite of how the systemd sysusers module works. Adding back the gitpuppet group is possible, but it causes some issues because we are using a gitpuppet user to sync the repo. [15:59:59] Do we need the gitpuppet group, i.e. what advantage does it bring over just specifiying which groups can access secrets, as of now that is only ops [16:06:45] that's a good question! to which I have no ad hoc answer(s), I'll have to dig a little in the history there [16:07:02] maybe jbond remembers [16:07:37] i ping jbond on why the group doesn't exist on the puppetservers, and he agreed that it was probably and oversight [16:07:42] pinged [16:08:42] this is the original commit, bringing it into existence for the puppetmasters, 5058d1476282e577eb63909a9cf3b782e333b964 [16:10:44] from what I can tell, people always edit secrets as root, so I'm not sure how this group was ever used [16:16:35] after poking around a bit, I tend to a agree, every is supposed to only edit the private repo as root, we never used the functionality, so I think we can actually discard the group [16:17:08] but let's ping Alex on task, given that he was the author of the original 2016 commit, maybe this was part of a bigger plan which never happened or so? [16:19:02] yeah will do, thanks for digging in a bit [16:25:17] I can't think of any reason for that group either, I tried to add it but as Jesse pointed out there is a little puppet mess to resolve [16:57:02] ftr i agree with the above [16:59:21] thanks jbond [17:42:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10614209 (10cmooney) >>! In T382017#10613705, @VRiley-WMF wrote: > Will be adding these into netbox shortly Cool I can see them there. FWIW I adde... [18:51:43] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Different BFD settings on direct connected links - https://phabricator.wikimedia.org/T387773#10614383 (10Papaul) bfd config removed from cr1/2-codfw on interface ae0 ` set protocols ospf area 0.0.0.0 interface ae0.0 interface-type p2p set protocols... [20:07:19] 10Mail, 06Infrastructure-Foundations, 06SRE: Consider Postfix as MTA for our MXes (and OTRS/Mailman/Phab) - https://phabricator.wikimedia.org/T232343#10614521 (10jhathaway) 05Open→03Resolved Postfix has replaced Exim for our inbound and outbound mail servers in production for some time now. Though th... [22:45:44] FIRING: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [22:55:44] RESOLVED: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting