[00:41:03] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10Dzahn) thanks! mgmt password updated using cookbook. [12:53:27] 10Traffic, 10Core Platform Team, 10Editing-team, 10Fundraising-Backlog, and 11 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10phuedx) [16:11:29] ema: please carefully review https://gerrit.wikimedia.org/r/c/operations/puppet/+/542153 https://i.kym-cdn.com/entries/icons/original/000/008/342/ihave.jpg [16:14:55] cdanis: How would it even get to the code you added? [16:15:10] oh [16:15:12] I can't read [16:15:20] you made me concerned I couldn't read! [16:15:21] SIMILAR CODE LOOKS SIMILAR [16:15:37] I was reading it as the same, but then my brain couldn't work out why it aligned differently on the right [16:15:40] (for the record, I'm still not convinced that I can) [16:15:55] It's not the first time I've made this mistake today [17:04:27] cdanis: hello! [17:04:49] cdanis: could you please add a test case? See modules/varnish/files/tests/text/31-blocked-nets.vtc [17:04:58] ema: ah thanks, will do! [17:05:08] I have to run an errand right now but will do so this afternoon [17:05:13] sounds good [17:06:50] cdanis: see ./modules/varnish/files/tests/README for how to run the tests on your machine :) [17:30:24] cdanis: you're also gonna need to change this https://github.com/wikimedia/labs-private/commit/fa6d1b76e25fd3741b70b27826defff0d91de363 (for the vtc tests) and the actual file in /srv/private for prod [17:32:15] yepyep ofc [19:04:42] hey Traffic, I'm investigating one task (https://phabricator.wikimedia.org/T233609) for our (Readers Web Team) project and I need your input. It would be amazing if we could hangout for like an hour and talk what's feasibile and what's totally off the table. /cc bblack vgutierrez ema sukhe [19:05:11] in short - we want to A/B test logged out users on some wikis [19:05:19] on desktop skin [19:08:15] hi raynor. I am around but I don't think I can be useful with this so you will have to wait for one of the other folks to reply (fyi, bblack is on vacation, it probably is evening for vgutierrez and ema) [19:08:39] I think ema is in CA at a conference? but also, at a conference :) [19:09:11] oh is he still there? and yeah :) [19:10:06] I don't need answers at this moment. It's even better if I can do that tomorrow (I'm in the EU timezone). [19:10:52] for now I'm trying to find someone who can answer my questions in the convenient (for that person) time. [19:11:10] it's more "let's plan to talk about that" instead "tell me now" :) [19:12:15] I'll ping ema and vgutierrez tomorrow EU morning. Thanks [19:13:48] :) [20:16:09] 10Traffic, 10Operations, 10observability: global HTTP (un)availability number, as reported in Frontend Traffic dashboard, is bogus - https://phabricator.wikimedia.org/T234567 (10fgiunchedi) +1 on at least a week's worth of data [20:33:45] ema: are timeouts after 30 seconds in text/01-w.wiki-shortener.vtc and text/07-backend-misspass_mangle-proper-caching.vtc expected...? [21:02:39] cdanis: mmh, nope. Which node are you targeting for testing? [21:03:00] ema: cp4027, just some random text node [21:03:20] right, yeah that should work [21:03:37] ah [21:03:43] maybe try a text node in eqiad [21:03:49] ah ok [21:04:22] not cp1075 though, which is ats :) [21:07:14] 0 tests failed, 0 tests skipped, 26 tests passed [21:07:17] on cp1077 [21:08:37] I'm trying to find a rationale explanation, but meanwhile \o/ [21:24:19] cdanis: lgtm, nice one [21:24:27] thanks for your help! [21:27:09] cdanis: rational excuse found, see sub misspass_mangle in modules/varnish/templates/text-backend.inc.vcl.erb [21:27:28] that code is executed only if (!bereq.http.X-Next-Is-Cache) [21:27:58] IOW only if we're in a DC where the next hop is the applayer [21:28:18] which isn't the case for ulsfo, but is for eqiad [21:28:57] ahhh [21:36:19] ema: okay I am about to add acl bot_blocked_nets (and additional comments) to the file in puppet private, then I will +2 and merge the puppet change itself [21:36:35] cdanis: ack! [21:36:36] ema: any need to do anything like disable puppet on all cps and test by hand on one host? [21:36:51] (AIUI, config reloads are always safe in varnish, right?) [21:36:58] cdanis: that's correct [21:37:08] ok I will proceed, thanks :) [21:37:32] cdanis: you can disable puppet to err on the side of caution if you really want (to avoid icinga spam in case of issues), but that's up to you [21:37:47] ahh, icinga spam for failing puppet runs is already fixed! [21:38:46] but it is not for failed vcl reloads :) [21:38:56] ah okay [21:43:40] ran puppet agent by hand on cp1077 and still looks good [21:43:53] nice! [21:43:57] thanks again! [21:44:02] I'll update the docs as well [21:44:03] thank you