[01:01:40] 06Traffic: ncmonitor shouldn't submit empty CRs to operations/dns repo - https://phabricator.wikimedia.org/T373780#10334301 (10BCornwall) 05In progress→03Resolved [01:11:07] 06Traffic, 13Patch-For-Review: main project domains are improperly added/evaluated in ncmonitor patches - https://phabricator.wikimedia.org/T374640#10334307 (10BCornwall) 05Open→03In progress [09:41:00] 06Traffic, 06Data-Platform, 06Data-Platform-SRE: 403 on http://dumps.wikimedia.org - https://phabricator.wikimedia.org/T379990#10334999 (10BTullis) [09:42:12] 06Traffic, 06Data-Platform, 06Data-Platform-SRE: 403 on http://dumps.wikimedia.org - https://phabricator.wikimedia.org/T379990#10335001 (10BTullis) p:05Triage→03Medium [10:57:24] vgutierrez: o/ when you have time/patience to go through https://gerrit.wikimedia.org/r/c/operations/puppet/+/1091597 again lemme know, it should encapsulate all your suggestions [10:57:44] sure.. I'm currently banging my head against some eBPF test [10:58:14] ahahhaha yes yes whenever you get out of any other project with more priority :) [10:58:29] I don't want to interfere with any self-nerd-snipe or similar [11:01:29] 06Traffic: Upgrade haproxy to 2.8.12 on cp hosts - https://phabricator.wikimedia.org/T379891#10335226 (10Fabfur) Currently deployed in uslfo and eqsin [11:04:18] elukey: depending on how paranoid you're you could leverage the cache_hosts list [11:04:25] but it looks good [11:05:07] vgutierrez: I chose mild paranoia, seems a good compromise atm [11:05:11] wdyt? [11:05:39] ah but can I get the subnets via puppet? [11:05:49] that would be awesome to avoid copying the raw subnets [11:06:06] we even got firewall rules available [11:06:25] this could be interesting, need to investigate more [11:06:32] maybe as second step [11:06:36] thanks a lot for the feedback [11:06:49] profile::wmcs::google_api_proxy is a good example [11:06:51] learned a lot on nginx, especially how location + if can be evil [11:07:37] their implementation isn't smart [11:07:41] <% @trusted_xff.each do |xff| -%> [11:07:41] set_real_ip_from <%= xff %>; [11:07:41] <% end -%> [11:08:03] so it directly gets the list of cache_hosts IPs and creates one line per IP [11:08:36] that works of course, but sorting them could lead to some kind of potential aggregation [11:08:41] so a long list yes [11:08:50] yeah... ~112 hosts at the moment IIRC [11:09:35] they get 224 lines given the IPv4/IPv6 duality [11:09:51] but given that a low-traffic service is IPv4 only you could ditch the IPv6 ones [11:11:24] I wasn't sure about this, if I could have assumed ipv4 only or not [11:12:20] if you keep IPv6 support there you're future proofing it [11:12:29] but right now IPv4 only would do the trick [11:16:02] ack ack [14:10:15] 10netops, 06Infrastructure-Foundations, 06serviceops, 07Kubernetes: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335773 (10ops-monitoring-bot) depool host wikikube-worker1290.eqiad.wmnet by a... [14:10:57] 10netops, 06Infrastructure-Foundations, 06serviceops, 07Kubernetes: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335776 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node star... [14:19:30] 10netops, 06Infrastructure-Foundations, 06serviceops, 07Kubernetes, 13Patch-For-Review: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335855 (10akosiaris) >>! In T379790#10330660, @cmooney w... [14:21:38] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10335878 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host cp7007.magru.wmnet with OS bullseye [15:15:38] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10336157 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host cp7007.magru.wmnet with OS bullseye completed: -... [15:52:46] 06Traffic: ncmonitor should have a dry-run option - https://phabricator.wikimedia.org/T380287 (10BCornwall) 03NEW [16:43:06] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10336729 (10Fabfur) As preliminary test before tomorrow's work, we reimaged cp7007 and verified all runs fine. It ran fine. [16:56:45] 06Traffic: ncmonitor should have a dry-run option - https://phabricator.wikimedia.org/T380287#10336793 (10BCornwall) p:05Triage→03Low [17:04:46] 06Traffic: Upgrade lshw on all DNS hosts - https://phabricator.wikimedia.org/T380295 (10Fabfur) 03NEW [17:09:37] 06Traffic: Upgrade lshw on all cp hosts - https://phabricator.wikimedia.org/T380295#10336834 (10ssingh) [17:29:58] 06Traffic, 10CX-cxserver, 10RESTBase Sunsetting, 07Essential-Work: Block RESTBase cxserver v1 endpoints in favor of the new endpoints - https://phabricator.wikimedia.org/T375616#10337009 (10akosiaris) a:03akosiaris I gather that we disable this at the CDN layer at any time, so I 'll be doing it this week... [17:47:59] 10netops, 06Infrastructure-Foundations, 06SRE: Consolidate Automation Templates for DC Switches - https://phabricator.wikimedia.org/T312635#10337207 (10Aklapper) a:05cmooney→03None @cmooney: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task as... [18:00:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10337359 (10cmooney) Ok. So I've tested the "[[ https://netbox.wikimed... [18:06:10] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru: installation tracking for hosts affected by magru re-shuffle - https://phabricator.wikimedia.org/T380307 (10RobH) 03NEW [18:06:12] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru: installation tracking for hosts affected by magru re-shuffle - https://phabricator.wikimedia.org/T380307#10337484 (10RobH) [18:06:17] 06Traffic, 10ops-magru, 06SRE, 13Patch-For-Review: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337485 (10RobH) [18:07:34] 06Traffic, 10ops-magru, 06SRE, 13Patch-For-Review: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337492 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7007.magru.wmnet with... [18:34:26] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337748 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye executed with... [18:34:56] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337749 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye [18:47:44] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337769 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye executed with... [18:48:15] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337770 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye [18:56:16] 06Traffic, 10CX-cxserver, 10RESTBase Sunsetting, 07Essential-Work: Block RESTBase cxserver v1 endpoints in favor of the new endpoints - https://phabricator.wikimedia.org/T375616#10337803 (10MSantos) >>! In T375616#10337009, @akosiaris wrote: > I gather that we disable this at the CDN layer at any time, so... [19:08:34] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337836 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye executed with... [19:08:59] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337838 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye [19:14:15] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337856 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye executed with... [19:15:28] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337867 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye [19:34:31] traffic folks, fyi i'm letting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1087615 slow-roll out to eqiad upload cps (also manually ran puppet on cp1101 to verify haproxy happiness) [19:35:42] thanks for confirming on one [19:35:45] gl [19:41:22] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10337988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7007.magru.wmnet with OS bullseye executed with... [20:34:16] 06Traffic, 10conftool, 13Patch-For-Review: Integrate requestctl haproxy rules into our TLS terminator - https://phabricator.wikimedia.org/T370745#10338188 (10CDanis) 05Resolved→03Open I'm reopening this to track the rollout of this feature beyond cp4044. [23:40:08] 10Domains: Park pay-for-edit and scam domains - https://phabricator.wikimedia.org/T380334 (10BCornwall) 03NEW