[09:17:34] 10SRE-tools, 10Spicerack: Puppet module hiera_lookup not working - https://phabricator.wikimedia.org/T378331 (10Volans) 03NEW p:05Triage→03High [09:23:35] elukey: maybe you can answer this supermicro question [09:23:54] on the Dell's I used to ssh to the idrac IP/mgmt hostname [09:24:04] and issue "console com2" to reach the virtual serial tty [09:24:12] is there an equivalent on supermicro? [09:24:23] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Puppet module hiera_lookup not working - https://phabricator.wikimedia.org/T378331#10266765 (10Devrepo) [09:24:48] topranks: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Supermicro_Documentation [09:24:57] doh :( [09:25:03] I should have thought to look first, thanks [09:25:07] it could be expanded though... [09:25:16] topranks: o/ I think that the page is very new, I had a similar one with those two commands [09:25:23] volans: I added 100% of my knowledge on that page :) [09:25:26] tried to ask to dcops several times, maybe we can create a task [09:25:32] see also the request from mo.ritz in -dcops ;) [09:25:57] elukey: +1 for the task [09:26:25] that worked fine anyway so thanks :) [09:32:07] they have a 195 page PDF on this, I'm currently reading though it to find the equivalent of "racadm getsel": https://www.supermicro.com/manuals/other/IPMI_Users_Guide.pdf [09:59:15] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional a6uto-conf IPv6 address - https://phabricator.wikimedia.org/T378335 (10cmooney) 03NEW p:05Triage→03Medium [10:11:23] there is an alert related to puppetboard.discovery.wmnet's puppet cert about to expire. I see that we use cfssl, ok for me to puppet destroy puppetboard.discovery.wmnet ? [10:26:48] sgmt [10:41:31] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional a6uto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267055 (10cmooney) [10:45:30] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional a6uto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267071 (10cmooney) It seems that changing the "ip token" command from "pre-up" to "up" in /etc/network/interfaces makes things work as expected. Re... [10:51:34] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional a6uto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267081 (10ayounsi) There might be some edge cases, but I think ideally we should disable the [[ https://sysctl-explorer.net/net/ipv6/autoconf/ | aut... [11:24:57] 10SRE-tools, 06Infrastructure-Foundations: redfish: minimum version support - https://phabricator.wikimedia.org/T328593#10267253 (10ayounsi) If I understand correctly, this task is about upgrading iDRAC to be able to upgrade iDRAC or other firmware more easily in the future. If that's the case I don't think w... [11:26:20] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional auto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267264 (10cmooney) [11:26:35] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional auto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267269 (10cmooney) [11:27:06] 10netops, 06Infrastructure-Foundations, 06SRE: Ganeti network config results in additional auto-conf IPv6 address - https://phabricator.wikimedia.org/T378335#10267251 (10cmooney) >>! In T378335#10267081, @ayounsi wrote: > There might be some edge cases, but I think ideally we should disable the [[ https://sy... [11:27:52] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to set up ganeti host network - https://phabricator.wikimedia.org/T378346 (10cmooney) 03NEW p:05Triage→03Low [11:48:29] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to set up ganeti host network - https://phabricator.wikimedia.org/T378346#10267435 (10Volans) Should it be part of the `sre.ganeti.addnode` cookbook or done at reimage time? [11:59:29] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to set up ganeti host network - https://phabricator.wikimedia.org/T378346#10267464 (10MoritzMuehlenhoff) >>! In T378346#10267435, @Volans wrote: > Should it be part of the `sre.ganeti.addnode` cookbook Mo, having this created (and rebooted) is... [14:18:49] hi XioNoX I'm reviewing this one https://groups.google.com/u/0/a/wikimedia.org/g/ops-maintenance/c/zzgzxeb7rk4 to determine if I need to add it to the maintenance calendar. AFAICS, there are no other sessions from eqiad to meta, though we have others from different DCs. Do I need to add an event to the calendar? [14:21:16] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: exception raised for "sre.dns.admin show" - https://phabricator.wikimedia.org/T378039#10267919 (10Volans) p:05Triage→03Medium [14:23:18] 07Puppet, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10observability: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853#10267925 (10elukey) p:05Triage→03Medium [14:26:18] tappof: you can skip those kind of notifications when they're just from IX peers [14:26:57] the majority of them don't notify us (and we don't notify them) for maintenance/outages [14:27:32] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Automate interface configuration for pfw firewalls using Netbox data - https://phabricator.wikimedia.org/T378070#10267928 (10cmooney) 05Open→03Resolved [14:31:36] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10267943 (10cmooney) >>! In T377381#10263676, @Dwisehaupt wrote: > @cmooney @Jclark-ctr There has been a re... [14:41:10] ok XioNoX, thanks [15:19:19] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10268212 (10jhathaway) p:05Triage→03Medium [15:29:08] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10268270 (10jhathaway) Thanks @revi, perhaps the `From:` header has changed since T272750, @Nahid any idea who on Trust & Safety manages this zendesk inst... [16:59:50] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10269002 (10Nahid) That would be @JAbrams. Let me also raise this in our team meeting tomorrow. [18:01:08] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10269530 (10revi) >>! In T378285#10268269, @jhathaway wrote: > Thanks @revi, perhaps the `From:` header has changed since T272750, Digging through my in... [18:19:34] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10269603 (10revi) Anyway I think possible solutions would be… 1. Route outgoing `wikimedia.org` mails through either Wikimedia infra (ie. the stuff you u... [19:02:00] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10269892 (10Jclark-ctr) @cmooney Sorry for delay i am good for nov 12th 100GBase-CWDM4 (green handle) w... [19:11:04] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10269922 (10jhathaway) Thanks for mentioning option (1), I was involved with re-setting that up for Qualtrics in T314815, and we could do something simila... [19:57:38] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC test - https://phabricator.wikimedia.org/T378285#10270193 (10revi) Heh! Didn't notice that — I was just reading the subject and descriptions (lol). Anyway: At least option 3 is possible. [support.zendes... [20:42:15] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10270349 (10Dwisehaupt) @cmooney @Jclark-ctr Wonderful, thanks for the flexibility. We should get confirmat...