[03:31:17] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10194593 (10Papaul) @ayounsi I will soon be setting up interfaces and assigning them to VLAN's. I wanted to know if we are... [03:42:55] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10194601 (10Papaul) @ayounsi @cmooney I have been working on the migration process and put together the proposal below. I also had a meetin... [04:01:01] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10194641 (10Papaul) [06:37:17] RhinosF1: The signup template is in: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/bitu/+/refs/heads/master/src/bitu/signups/templates/signups_signup.html [06:38:40] slyngs: is some of the interface messaging defined elsewhere [06:39:03] Ctrl+f doesn't find the bit of text I'm looking for in that page [06:39:23] The menu is in a separate template, as is part of the layout. Which bit are you looking for? [06:43:17] slyngs: Your email must be unique. If you are setting up an account for a bot, please use a dedicated email address or an address suffix, e.g. user+bot@example.com (if you mailserver supports this). [06:43:41] which has a typo in which was annoying me last night so I was going to make a patch but couldn't find that text anywhere [06:43:47] :-) [06:44:22] that is in "signups/models.py" is is part of the helper text for that field [06:44:55] Line 68 [06:44:58] I will have you a very tiny patch in a few moments [06:45:09] Thank you :-) [06:46:10] slyngs: https://gerrit.wikimedia.org/r/c/operations/software/bitu/+/1077241 [06:55:18] Thanks, that is annoying once you see it :-) [06:55:59] Yup, thanks for the +2 :) [08:09:24] 10netbox, 06Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169#10194839 (10ayounsi) Thinking out loud I'm wondering if we could/should add an ASN (multi-)object(s) custom field to prefixes. The... [08:21:48] FIRING: PuppetFailure: Puppet has failed on bast1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:40:06] 10netbox, 06Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169#10194890 (10cmooney) That's not a bad idea. Thinking it through it would appear to be a fairly robust way, being multi-field can... [09:03:12] hey folks, as FYI I just depooled aux-k8s-{ctrl,worker}1001 from AUX k8s, they will be decommed if nothing explodes (new VMs are already replacing them) [09:06:48] RESOLVED: PuppetFailure: Puppet has failed on bast1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:27:14] Refish question for elukey or volans, is it possible to get the list of possible values for `BIOS.Setup.1-1` for a given host (dell or supermicro) ? https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/hosts/provision.py#L203C14-L203C28 [09:28:17] no, you can dump the existing values from a real host [09:28:34] but dependening on generation, idrac/bmc version and settings they might differ [09:28:43] oh that's a good start, is there doc on how to do that ? [09:28:54] for example for dell if bootmode is bios or uefi a lot of keys change [09:28:58] not the values, the keys [09:30:30] yeah, the end goal is to manually or using a cookbook to set it to uefi and httpboot [09:31:03] r = spicerack.redfish('sretest1003') [09:31:05] scp = r.scp_dump() [09:31:28] scp.components['BIOS.Setup.1-1'].keys() [09:31:48] see https://doc.wikimedia.org/spicerack/master/api/spicerack.redfish.html#spicerack.redfish.DellSCP [09:38:56] XioNoX: clear enough? ^^^ [09:39:34] volans: thanks yes it looks straightforward but I'm getting an error [09:39:44] so I was checking if t was something stupid on my side [09:39:49] this is for dell [09:39:58] SCP is a dell-specific tech [09:40:19] which error are you getting? [09:40:20] yeah I'm testing it on sretest1002 which is dell [09:40:24] https://www.irccloud.com/pastebin/ygNAshML/ [09:40:52] using a dryrun spicerack instance if it matters [09:41:48] trying [09:41:55] WFM [09:42:17] maybe is the dry-run, not 100% sure, should check the code [09:43:42] volans: yep, it was the dry-run [09:43:56] feel free to open a task to luca ;) [09:58:21] XioNoX: are you folks using a task or something that I can read? Curious :) [09:58:30] I checked https://phabricator.wikimedia.org/T373519 but didn't find news [10:05:06] elukey: we use a messy etherpad :) I'm stepping out for lunch, I can share it after if you don't mind being spoiled [10:06:49] anytime :) [11:24:44] elukey: https://etherpad.wikimedia.org/p/bios-be-gone [12:38:35] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195673 (10aborrero) >>! In T375847#10187153, @cmooney wrote: > @aborrero the network assignment is incorrect also. > [[ https://netbo... [12:43:50] thanks! [12:46:16] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195699 (10aborrero) I guess next bits to test with neutron would be to enable north-south traffic, meaning working on these two ticke... [12:49:58] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195719 (10cmooney) >>! In T375847#10195699, @aborrero wrote: > I guess next bits to test with neutron would be to enable north-south... [12:58:24] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10195740 (10cmooney) At a high level I think we need to: * Create an aggregate policy on //cloudsw1-b1-codfw// to generate 2a02:ec80:a100::/48 if par... [14:21:04] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10196144 (10nisrael) @Reedy following up on this. Any update on what could be causing the emails to direct to that inbox? [14:23:20] reposting from #wikimedia-tech [14:23:22] 10:01:18 Hello everyone. My name is William Rai. I recently joined as a software engineer on Android team. I created a Wikimedia dev account using this link https://idm.wikimedia.org/signup/. After activating, it displayed this message "Thank you for activating your account. Your account is now confirmed and is currently being created. Please allow for up [14:23:24] 10:01:18 to a few minutes for the creation to complete, after which you will be able to log in using your selected password." Based on the message I should have received the login info. I made this account yesterday around 4 PM and so far I have not received any email about the account. [14:25:32] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10196151 (10Reedy) No, I don't have any access to check into this. It was just unclear from the report where she was actually (incorrectly) recieving them. [14:41:08] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10196186 (10nisrael) What would be next steps for investigating this? [14:48:39] cdanis: probably worth to suggest a task creation [14:59:25] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10196267 (10Aklapper) Likely #Infrastructure-Foundations ( https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering#Infrastructure_Foundation... [15:25:29] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10196352 (10nisrael) Sounds good. Anything I can do from my end to help? Will Infrastructure team be able to continue discussing in this phab? [15:39:56] elukey: turns out kamila_ reached out to the user about it, all resolved and not a systematic issue [15:40:45] super [15:48:33] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10196558 (10aborrero) here is a proposal: * 2a02:ec80:a100:fe01::/64 - cr1-codfw uplink * 2a02:ec80:a100:fe02::/64 - cr2-codfw uplink * 2a02:ec80:a10... [16:03:36] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10196649 (10ayounsi) No interface range as each switch will be independent. [16:04:17] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10196661 (10Papaul) Thanks [20:16:21] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 22.4R3-S2 - https://phabricator.wikimedia.org/T369504#10197756 (10Papaul) The recommended Junos version for srx300 is 23.4R2-S2 as for 2024-9-10. Are going for version 23 or 22? [22:28:39] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10198161 (10cmooney) That seems fine to me @aborrero thanks! [23:39:50] 10netops, 06Infrastructure-Foundations, 06SRE-OnFire, 10Sustainability (Incident Followup): Juniper: regularly run `request system configuration rescue save` - https://phabricator.wikimedia.org/T376005#10198393 (10Dzahn)