[09:05:15] 10netops, 06Infrastructure-Foundations, 06SRE: ToR server-move Netbox script adding ".0" to end of interface names - https://phabricator.wikimedia.org/T374024#10245434 (10Aklapper) @cmooney: Could you please answer the last comment? Thanks in advance! :) [09:14:47] 10netops, 06Infrastructure-Foundations, 06SRE: ToR server-move Netbox script adding ".0" to end of interface names - https://phabricator.wikimedia.org/T374024#10245492 (10cmooney) 05Open→03Resolved Apologies, yes all this is complete, resolving. [09:15:11] 06Traffic, 10conftool, 07Epic: Create simple web view of requestctl status - https://phabricator.wikimedia.org/T371782#10245505 (10Joe) The basic read-only interface is published, we can solve this task and open a new one for the read-write interface [09:15:58] 06Traffic, 10conftool, 07Epic: Create simple web view of requestctl status - https://phabricator.wikimedia.org/T371782#10245511 (10Joe) 05In progress→03Resolved [09:19:06] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.4 Improve our existing tooling to allow quicker reaction times to ongoing attacks. - https://phabricator.wikimedia.org/T369480#10245521 (10Joe) Most of the user stories for this epic have been implemented. Some were out of scope for this quarter's hypothe... [09:19:52] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.4 Improve our existing tooling to allow quicker reaction times to ongoing attacks. - https://phabricator.wikimedia.org/T369480#10245522 (10Joe) 05Open→03Resolved a:03Joe [09:28:52] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.7 Roll out a user-friendly web application that enables assisted editing and creation of requestctl rules - https://phabricator.wikimedia.org/T377699 (10Joe) 03NEW [09:29:49] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.7 Roll out a user-friendly web application that enables assisted editing and creation of requestctl rules - https://phabricator.wikimedia.org/T377699#10245553 (10Joe) p:05Triage→03High [09:31:46] 06Traffic, 10conftool, 10Sustainability (Incident Followup): Make it easier to create a new requestctl object - https://phabricator.wikimedia.org/T310009#10245561 (10Joe) [09:31:47] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.7 Roll out a user-friendly web application that enables assisted editing and creation of requestctl rules - https://phabricator.wikimedia.org/T377699#10245562 (10Joe) [09:31:48] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.4 Improve our existing tooling to allow quicker reaction times to ongoing attacks. - https://phabricator.wikimedia.org/T369480#10245563 (10Joe) [09:32:54] 06Traffic, 10conftool, 07Epic: Coexistence of the requestctl CLI tool and of the web interface - https://phabricator.wikimedia.org/T374723#10245566 (10Joe) [09:32:55] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.7 Roll out a user-friendly web application that enables assisted editing and creation of requestctl rules - https://phabricator.wikimedia.org/T377699#10245567 (10Joe) [09:32:56] 06Traffic, 10conftool, 07Epic: Create simple web view of requestctl status - https://phabricator.wikimedia.org/T371782#10245568 (10Joe) [09:34:45] 06Traffic, 10conftool, 07Epic, 13Patch-For-Review: Deprecate sync, add apply command to requestctl - https://phabricator.wikimedia.org/T376877#10245569 (10Joe) 05Open→03In progress p:05Triage→03High [09:34:45] 06Traffic, 10conftool, 06Tech-Docs-Team, 07Documentation: Create new content structure for requestctl tool documentation - https://phabricator.wikimedia.org/T372095#10245572 (10Joe) [09:34:48] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.7 Roll out a user-friendly web application that enables assisted editing and creation of requestctl rules - https://phabricator.wikimedia.org/T377699#10245573 (10Joe) [09:34:50] 06Traffic, 10conftool, 07Epic: [EPIC] FY 24/25 WE 4.3.4 Improve our existing tooling to allow quicker reaction times to ongoing attacks. - https://phabricator.wikimedia.org/T369480#10245574 (10Joe) [12:40:51] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10246081 (10aborrero) [12:44:43] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10246085 (10aborrero) 05Open→03In progress Let me check what is left to be done here. [13:46:32] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10246381 (10CDanis) 05Open→03Resolved a:03CDanis ` root@db1169:~# ss -tr | grep :mysql ESTAB 0 945 db1169.eqiad.wmnet:mysql 10-67-163... [13:49:11] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10246391 (10ssingh) Hi @RobH: Thanks for writing the instructions in the doc above. The hostnames and other information looks good. Is there any thing pen... [13:53:17] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10246403 (10cmooney) p:05Triage→03Medium What's the current thinking here? Are we agreed to widen both allocations to /17... [14:25:58] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10246585 (10Jgreen) @cmooney Dallas did a survey of existing servers at eqiad, and none of them have 10G in... [14:31:50] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10246614 (10cmooney) >>! In T377381#10246582, @Jgreen wrote: > @cmooney Dallas did a survey of existing ser... [14:33:12] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10246616 (10cmooney) [14:44:04] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-tools, 13Patch-For-Review: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485#10246717 (10Volans) a:05Volans→03None [14:53:26] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10246786 (10akosiaris) Good question. Let me add some data points. We currently use: ` root@deploy1003:~# kube_env admin eqia... [15:10:43] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10246886 (10Jgreen) > That's a bit of a shame in some ways but no problems We'll get there next procuremen... [15:46:24] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10247069 (10RobH) I still need to finish up the directions and will ping everyone to proof them again, but with all the ordering this week I just got a bi... [15:47:01] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10247070 (10Papaul) [15:47:22] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10247072 (10ssingh) Thanks @RobH. Please let us know if Traffic's input is required or if we can help. [17:10:04] hi folks, any concerns if I were to configure two new LVS services at some point in the next two hours? (configuring both at the same time in order to minimize pybal restarts) [17:11:44] swfrench-wmf: if anything, feel free to do them independently as that might be better (testing, isolating any issues if any). but no issues from our end [17:13:38] sukhe: ah, interesting! it did not occur to me that breaking them up at the expense of more restarts would actually be lower risk. [17:13:57] swfrench-wmf: I am not sure if there is a right answer but that's at least what we do :] [17:15:25] sukhe: got it, sounds good - I'll break up my patches and get back to you once I'm ready to start [17:15:34] <3 [17:53:07] sukhe: alright, I have my patches together and I'll get started on the first of the two services shortly [17:53:34] ok! [18:25:04] sukhe: any preference on how long to wait between services (first of two is done) [18:26:30] swfrench-wmf: since you used the cookbook (thank you!) and if things are looking good, starting with the backup without any wait should be totally fine [18:28:56] sukhe: great, yeah no issues encountered. I'll move ahead to the next service and follow the same sequence (i.e., starting with the backup). thanks for making this straightforward with the cookbook and all :) [18:30:50] swfrench-wmf: credit for the cookbook goes to the other fine folks here; I merely added the aliases to cumin and shepherd their use :) [18:30:59] but yeah, the cookbook for this is quite nice indeed [18:31:29] I mean, having the aliases available is a huge help :) [19:14:38] all done, after a brief disruption to wait for a deployment to pass [19:17:57] thanks! and for the clean run as well.