[07:38:19] 06Traffic: ncmonitor shouldn't submit empty CRs to operations/dns repo - https://phabricator.wikimedia.org/T373780 (10Vgutierrez) 03NEW [07:38:40] 06Traffic: ncmonitor shouldn't submit empty CRs to operations/dns repo - https://phabricator.wikimedia.org/T373780#10109316 (10Vgutierrez) p:05Triage→03Medium [08:15:55] 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095#10109414 (10ABran-WMF) | es2032| es1 standalone| |es2031 |es2 standalone| |db2207 |s2 candidate master| |db2138| s2| |db2125... [08:49:40] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109497 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes200... [08:50:31] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes202... [09:02:30] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109565 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [09:03:14] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109571 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [10:06:26] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109751 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [10:21:19] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [11:02:03] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109838 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [11:02:06] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109839 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [11:02:38] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109841 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [11:20:26] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109865 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [11:21:22] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109867 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [11:22:28] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109870 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [11:23:38] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2387 to w... [11:31:03] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109917 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2388 to w... [11:31:26] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109920 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2389 to w... [11:32:56] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109921 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2386 to w... [11:38:26] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10109948 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2386 to w... [12:03:44] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10110036 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [12:05:03] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10110037 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2386 to w... [12:17:35] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10110056 (10Nux) I think you might need to check your stats based on UserAgent (if that is possible). You mig... [12:21:18] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10110083 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [13:05:42] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10110260 (10akosiaris) [13:21:18] 10netops, 06Infrastructure-Foundations, 06SRE: EX4600 does not support class-of-service 'port scheduling' - https://phabricator.wikimedia.org/T373594#10110339 (10cmooney) Despite this[[ https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/concept/cos-qfx-series-support-by-qf... [13:25:57] hey y'all :) is the regex_map in ATS config some special flavour of regex? I'm trying to figure out why urls like https://en.wikipedia.org/w/rest.php and https://en.wikipedia.org/api/ end up on mw-web instead of mw-api-ext. The regex_map target is target: 'http://(.*)/(w/rest.php|api/)' which I would think should work and redirect to mw-api-ext correctly (L435 of backend.yaml) [13:27:03] or maybe it needs its \ escaped [13:27:23] claime: https://www.lua.org/pil/20.1.html [13:28:01] also ordering of the backend.yaml file matters [13:28:39] afaict looking at the server config directly, we end up in the catchall that goes to mw-web [13:28:50] yep [13:29:42] 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095#10110383 (10Ladsgroup) Noting that es2032 is a "perceived master" (since dbctl requires a master) so you can't just depool i... [13:30:39] forget the lua regex confusision [13:30:43] *confusion [13:30:50] regex_remap uses PCRE [13:31:06] ok so I need to escape the backslashes then [13:31:09] cool [13:31:13] yep [13:31:59] the dot as well [13:32:07] afaict only those inside the capture group though? since the action api regex map right under works correctly [13:35:37] yep.. schema doesn't need to be escaped [13:36:34] cause "When the regular expression is matched, only the URL path + query string is matched" [13:41:20] 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095#10110436 (10ABran-WMF) >>! In T373095#10110383, @Ladsgroup wrote: > Noting that es2032 is a "perceived master" (since dbctl... [13:43:55] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10110447 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2066.codfw.... [13:45:46] claime: so, something like `(w\/rest\.php|api\/)`? [13:46:23] yep, that's what I went with, I'm sending the patch in a minute [13:56:37] claime: hmm that won't work [13:57:24] so what we are currently using is regex_map, not the regex remap plugin (naming is hard?) [13:57:55] from regex_map documentation (https://docs.trafficserver.apache.org/en/9.2.x/admin-guide/files/remap.config.en.html#regular-expression-regex-remap-support) --> Only the host field can contain a regex; the scheme, port, and other fields cannot. For path manipulation via regexes, use the Regex Remap Plugin. [13:58:22] claime: so right now the easiest way would be adding an additional entry for /api/ [13:58:25] ah, then I'll just duplicate the config and match one and the other [13:58:27] yep [13:58:32] sgtm [14:26:10] claime: are you submitting that one or should I? [14:27:04] I can do it if you +1 it [14:27:10] sure [14:27:31] disable puppet on cache::text, enable on one, test it there, then enable puppet everywhere and let it run? [14:34:30] claime: sure.. I haven't seen the gerrit CR though [14:35:03] vgutierrez: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1070032 [15:09:18] hmmm it worked a couple times for /w/rest.php but now seems to send back to mw-web, didn't work at all for /api/ [15:10:08] testing with `curl -v --connect-to en.wikipedia.org:443:localhost https://en.wikipedia.org/w/rest.php 2>&1 | grep server:`, `curl -v --connect-to en.wikipedia.org:443:localhost https://en.wikipedia.org/api/ 2>&1 | grep server:` and `curl -v --connect-to en.wikipedia.org:443:localhost 'https://en.wikipedia.org/w/rest.php/v1/search/page?limit=5&q=test' 2>&1 | grep server:` [15:10:59] on cp2027 [15:15:34] https://phabricator.wikimedia.org/P68532 [15:23:39] when I vary the search query I do end up on mw-api-ext, so I guess it reuses connections until the paths actually go stale? [15:25:27] * vgutierrez looking [15:26:18] ty <3 [15:29:04] vgutierrez@cp2027:~$ curl -s -v -H 'Host: en.wikipedia.org' http://en.wikipedia.org:3128/api/ 2>&1 |grep "server:" [15:29:32] that shows mw-api-ext consistently [15:29:58] you don't need --connect-to on cp hosts.. the VIP is on the loopback interface BTW [15:30:26] so ATS is behaving as expected [15:30:29] let's see what varnish is doing [15:31:39] vgutierrez@cp2027:~$ curl -s -v -H 'Host: en.wikipedia.org' https://en.wikipedia.org/api/ 2>&1 |egrep "server:|x-cache-status" [15:31:39] < server: mw-web.codfw.main-6d5c566995-58mg5 [15:31:39] < x-cache-status: hit-front [15:31:52] claime: you're getting a cached response from varnish [15:32:49] claime: on ATS both /api/ and /w/rest.php go to mw-api-ext [15:35:43] claime: also.. you're testing on one of the DCs where we haven't migrated to single backend yet, so a varnish request isn't guaranteed to hit ATS on the same cp server [15:36:12] oh yeah, that would explain the inconsistency then [15:36:44] because I just did two varnish requests with just a query param changed, both x-cache-status: pass, but one ended up on mw-api-ext and the other on mw-web [15:37:07] check x-cache header [15:37:27] and you should see one hitting cp2027 and the other one another cp server in codfw [15:37:47] < server: mw-web.codfw.main-6d5c566995-w2tdp [15:37:49] < x-cache: cp2033 miss, cp2027 pass [15:37:51] yeah [15:37:53] exactly [15:38:21] ok, then I can go ahead and re-enable puppet on the rest of the cache::text nodes [15:38:29] +1 [15:38:42] thanks, caching layer is still a little confusing to me x) [15:38:47] I've taken *notes* [15:39:49] that will shift some traffic around, I'll keep an eye on dashboards [15:39:53] thanks again for all the help [15:40:18] no problem [17:07:20] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE: Issues reimaging kubernetes workers due to package user issues in systemd-timesyncd - https://phabricator.wikimedia.org/T373819 (10hnowlan) 03NEW [17:13:54] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10111061 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [17:15:13] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10111066 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [17:16:49] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10111067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [17:19:14] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10111068 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [21:01:56] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10111308 (10Krinkle) 05Resolved→03Open This is broken, again, after {T364400}: >>! In T364400#10110727, @gerr... [21:08:56] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10111318 (10Krinkle) The patch also appears to have an off-by-one error. Notice how https://en.wikipedia.org/api/i...