[03:48:46] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10252637 (10Papaul) 05Open→03Resolved This is complete [03:48:53] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10252634 (10Papaul) 05Open→03Resolved This is complete. [08:41:57] 07Puppet, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10observability: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853#10253093 (10MatthewVernon) does `megacli` work? That's the tool (from the `megacli` package) that I... [08:51:54] 07Puppet, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10observability: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853#10253127 (10jcrespo) Nope, megacli doesn't work. That's the one option I tried first, before going o... [08:55:20] 07Puppet, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10observability: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853#10253136 (10jcrespo) >>! In T377853#10251006, @jcrespo wrote: > perccli and storecli are not exactly... [09:01:33] 07Puppet, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10observability: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853#10253146 (10MatthewVernon) Perhaps relevantly, I was screenshotting the BMC storage page on another... [09:12:37] 10CAS-SSO, 06Infrastructure-Foundations: Clean up hiera data for CAS installation on cloud - https://phabricator.wikimedia.org/T377925 (10SLyngshede-WMF) 03NEW [09:13:20] 10CAS-SSO, 06Infrastructure-Foundations: Clean up hiera data for CAS installation on cloud - https://phabricator.wikimedia.org/T377925#10253184 (10SLyngshede-WMF) p:05Triage→03Medium [09:13:21] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10253170 (10cmooney) >>! In T377381#10252289, @Jclark-ctr wrote: > @cmooney Step 1: Firewall Installation... [11:21:56] 10CAS-SSO, 06Infrastructure-Foundations: Create Redis database for IDP-Test - https://phabricator.wikimedia.org/T377937 (10SLyngshede-WMF) 03NEW [11:22:03] 10CAS-SSO, 06Infrastructure-Foundations: Create Redis database for IDP-Test - https://phabricator.wikimedia.org/T377937#10253586 (10SLyngshede-WMF) p:05Triage→03Low [11:54:16] 10CAS-SSO, 06Infrastructure-Foundations: Create Redis database for IDP-Test - https://phabricator.wikimedia.org/T377937#10253659 (10jijiki) You may grab a db here [[ https://wikitech.wikimedia.org/wiki/Redis | here]]. And yes, the major drawback of our redis infra is that, we can't auto failover to a secondar... [11:54:19] 10netbox, 10ChangeProp, 06collaboration-services, 06Infrastructure-Foundations, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10253657 (10jijiki) [12:11:34] 10netbox, 10ChangeProp, 06cloud-services-team, 06collaboration-services, and 11 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10253695 (10jijiki) [12:34:29] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10253766 (10aborrero) 05In progress→03Stalled Turns out, to enable PTR creation support, per {T377740} we would need to eit... [12:50:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10253808 (10ayounsi) a:05ayounsi→03Papaul [12:50:47] 10netops, 06Infrastructure-Foundations, 06SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028#10253811 (10ayounsi) 05Open→03Stalled [12:51:34] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Add Dell switches support to Homer/Cookbooks - https://phabricator.wikimedia.org/T320638#10253832 (10ayounsi) 05Open→03Stalled a:05ayounsi→03None [12:52:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10253836 (10ayounsi) a:03Papaul [13:35:02] 10netbox, 10ChangeProp, 06cloud-services-team, 06collaboration-services, and 11 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10254048 (10bking) Forgive the drive-by comment, but at the 6-month anniversary of this ticket, it might... [13:45:09] 10SRE-tools, 06Data-Persistence-SRE, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: mysql_legacy: SQL query quote escape - https://phabricator.wikimedia.org/T376712#10254082 (10ABran-WMF) 05Open→03Declined see T368881#10254014 [13:50:16] 10SRE-tools, 06Data-Persistence-SRE, 06DBA, 06Infrastructure-Foundations, and 2 others: mariadb: systemctl status accessor in mysql_legacy - https://phabricator.wikimedia.org/T377129#10254105 (10ABran-WMF) 05Open→03Resolved code is implemented, needs to be tested under T374191 [14:23:52] 10CAS-SSO, 06Infrastructure-Foundations, 06SRE Observability, 10Release-Engineering-Team (Radar): Document how to authenticate a bot account through CAS-SSO - https://phabricator.wikimedia.org/T377372#10254313 (10lmata) Hi @bd808 during our team sync we discussed this and dont have a good answer. This feel... [15:23:12] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254615 (10cmooney) [15:24:01] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254623 (10cmooney) [15:41:51] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254748 (10cmooney) [15:55:48] topranks: fyi, now LibreNMS have a "transceiver" page: https://librenms.wikimedia.org/device/device=1/tab=port/port=11611/view=transceiver/ [15:56:45] ooh nice [15:57:10] that definitely seems an easier way to navigate to the graph for historical values than the device-wide 'sensors' tab [17:00:43] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network eleemnts from Netbox - https://phabricator.wikimedia.org/T377996 (10cmooney) 03NEW p:05Triage→03Medium [17:01:19] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10255334 (10cmooney) Just a note to say that fundraising no longer use any VM infra, so every assigned IP I believe belongs to just a single server. [17:01:22] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network eleemnts from Netbox - https://phabricator.wikimedia.org/T377996#10255335 (10cmooney) [17:01:31] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10255336 (10cmooney) [17:01:47] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10255340 (10cmooney) [17:02:20] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10255341 (10cmooney) [17:03:13] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10255344 (10cmooney) [18:26:32] jhathaway: thanks, EFI was just something I've let installers hide from me for years now [18:27:43] cdanis: of course, I enjoyed the blog post as well, Lennart's writing is usually pretty good [18:27:47] yeah [18:29:02] cdanis: do you happen to know if we have any docs on when we choose software raid vs hardware raid? [18:29:25] as best as I know that is generally up to the service owner [18:30:07] I'm a pretty bad person to ask about the procurement side of things here though [18:31:22] no worries, I saw some docs on benchmarks we ran, but couldn't dig up anything else [18:37:13] yeah I'd probably ask Willy and also someone from DP [18:37:46] nod [19:25:57] 10Mail, 06Infrastructure-Foundations, 06SRE: postfix mx puppetry - https://phabricator.wikimedia.org/T325395#10255974 (10jhathaway) 05Open→03Resolved [19:26:10] 10Mail, 06Infrastructure-Foundations, 06SRE: Postfix MTA Profile - https://phabricator.wikimedia.org/T325398#10255976 (10jhathaway) 05Open→03Resolved [19:27:36] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mta-outbound-infra - https://phabricator.wikimedia.org/T325402#10255981 (10jhathaway) 05Open→03Invalid architecture was dropped in favor of only having mx-in and mx-out hosts. [19:27:44] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-out - https://phabricator.wikimedia.org/T325407#10255984 (10jhathaway) 05Open→03Resolved [19:28:14] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mta-inbound-infra - https://phabricator.wikimedia.org/T325401#10255978 (10jhathaway) 05Open→03Invalid architecture was dropped in favor of only having mx-in and mx-out hosts. [19:29:00] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-in - https://phabricator.wikimedia.org/T325406#10255987 (10jhathaway) 05Open→03Resolved [19:37:42] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10Znuny: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160#10256013 (10Dzahn) Thank you @jhathaway, cool. I think we... [19:48:53] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10Znuny: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160#10256047 (10Dzahn) a:05Dzahn→03None [19:49:55] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10Znuny: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160#10256044 (10Dzahn) 05Open→03Resolved a:03Dzahn `... [20:28:26] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021 (10jhathaway) 03NEW [20:28:35] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021#10256211 (10jhathaway) p:05Triage→03Medium [20:29:19] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-in-lists - https://phabricator.wikimedia.org/T325404#10256214 (10jhathaway) p:05Low→03Medium [20:29:31] 10Mail, 06Infrastructure-Foundations, 06SRE: MTA Provisioning - https://phabricator.wikimedia.org/T325403#10256235 (10jhathaway) [20:29:32] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-in-lists - https://phabricator.wikimedia.org/T325404#10256233 (10jhathaway) [20:29:33] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021#10256234 (10jhathaway) [20:31:08] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-out-lists - https://phabricator.wikimedia.org/T325405#10256236 (10jhathaway) [20:31:13] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-out-lists - https://phabricator.wikimedia.org/T325405#10256239 (10jhathaway) [20:31:16] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021#10256240 (10jhathaway) [20:31:19] 10Mail, 06Infrastructure-Foundations, 06SRE: MTA Provisioning - https://phabricator.wikimedia.org/T325403#10256241 (10jhathaway) [20:33:07] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim null client config with a Postfix null client config - https://phabricator.wikimedia.org/T325408#10256255 (10jhathaway) [20:36:09] 10Mail, 06Infrastructure-Foundations, 06SRE: MTA provisioning - https://phabricator.wikimedia.org/T325403#10256261 (10jhathaway) [20:36:17] 10Mail, 06Infrastructure-Foundations, 06SRE: MTA provisioning - https://phabricator.wikimedia.org/T325403#10256265 (10jhathaway) 05Open→03Resolved [20:45:21] 10Mail, 06Infrastructure-Foundations, 06SRE: Decom Exim based mx{1001,2001}.wikimedia.org - https://phabricator.wikimedia.org/T325409#10256361 (10jhathaway) [20:45:26] 10Mail, 06Infrastructure-Foundations, 06SRE: Decom Exim based mx{1001,2001}.wikimedia.org - https://phabricator.wikimedia.org/T325409#10256362 (10jhathaway) 05Open→03Resolved [20:48:48] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256380 (10cmooney) [20:51:00] 10Mail, 06Infrastructure-Foundations, 06SRE: Integration tests - https://phabricator.wikimedia.org/T358355#10256386 (10jhathaway) [20:58:22] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256417 (10cmooney) [21:00:31] 10Mail, 06Infrastructure-Foundations, 06SRE: Integration tests - https://phabricator.wikimedia.org/T358355#10256420 (10jhathaway) 05Open→03Resolved They are still a bit rough in places, but resolving for now: https://gitlab.wikimedia.org/jhathaway/mx-tests [21:02:29] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256437 (10Jclark-ctr) @cmooney all cables have been connected for Step 2: Initial cabling for the new de... [21:05:17] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256451 (10cmooney) >>! In T377381#10256437, @Jclark-ctr wrote: > @cmooney all cables have been connected... [21:10:29] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on VRTS servers with Postfix - https://phabricator.wikimedia.org/T378028 (10jhathaway) 03NEW [21:10:41] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on VRTS servers with Postfix - https://phabricator.wikimedia.org/T378028#10256498 (10jhathaway) p:05Triage→03Low [21:12:11] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on phabricator servers with Postfix - https://phabricator.wikimedia.org/T378029 (10jhathaway) 03NEW [21:12:36] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim on phabricator servers with Postfix - https://phabricator.wikimedia.org/T378029#10256514 (10jhathaway) p:05Triage→03Low [23:13:04] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: exception raised for "sre.dns.admin show" - https://phabricator.wikimedia.org/T378039#10256848 (10Dzahn) [23:39:58] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: exception raised for "sre.dns.admin show" - https://phabricator.wikimedia.org/T378039#10256892 (10ssingh) Thanks for filing this task! It's a known issue as documented in T365454#10179477 as well. That being said and in the meantime, I am curious to hear if... [23:44:26] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: exception raised for "sre.dns.admin show" - https://phabricator.wikimedia.org/T378039#10256896 (10Dzahn) >>! In T378039#10256892, @ssingh wrote: > That being said and in the meantime, I am curious to hear if you have a suggestion on how to improve this text....