[09:40:29] good morning again! I'm back for another try... can anyone here give a +1 on gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1354040 , be around while I deploy it, and then roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1356371 ? [09:41:09] duesen: I am here [09:41:42] what time would you lik ethe puppet change merged and rolled out? [09:42:00] also, let me know if letting it roll (~30') is ok [09:43:09] hey effi! the puppet change should not go outbefore the gateway deployment. Doing so would break static content under /api/. I don't think anyone is replying on that, so it wouldn't be a big deal. But still, I'd like to do them in the proper order. [09:43:31] s/replying/relying/ [09:43:37] I need another coffe [09:45:58] for the gateway deployment, I just need someone to be around, just in case something goes wrong. I'm confident in the change in behavior, and it has tests that will run against staging, to make sure it works. But check around the edges, please. traditionally i missed bumping the chart version - though in this case, it shouldn't be needed. [09:47:36] effie: the trafficserver change has a merge conflict... can you resolve that? i didn't make that patch, and I don't have the repo checked out... and I never touched ATS :D [09:51:35] I can take care of this [09:53:04] I will be around for the next ~3.5 hrous, what I would like is to know when you'd like to rollout 1354040 [09:53:19] and after this, how soon I should take care of the puppet stuff [09:54:23] I was hoping to deploy the gateway change in 20 or 30 minutes. Can you give me a +1 on that? [09:55:08] already did, let's crack on [09:57:07] and I will rebase the puppet patch [09:57:33] Ok, I'll be on it asap. I'll ping you when I start deploying. [09:57:58] no need, just ping me if something goes wrongly :) [09:58:30] and when I am good to start deploying my bit [10:17:20] effie: I'm seeing an unexpected block of lines being added in the diff for staging. To me it looks like https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1368714 was never merged into staging, so I would be applying that as well. Should be harmless, can you confirm? [10:18:09] I'm a little confused that I'm not seeing that on https://integration.wikimedia.org/ci/job/helm-lint/37595/console [10:18:21] That should be the same diff I see on the console, right? [10:18:52] mmm let me have a thing [10:18:57] think* [10:18:57] The relevant stanza is name: edit-analytics_edits_per_editor_v3 [10:19:41] if it is staging, we should not block ourselves on it [10:20:33] duesen: this is expected, the other change was merged previously [10:20:48] the fact that it was not rolled out to production, is a different story [10:20:53] effie: well, i don't know if that block is also missing from the prod cluster. it's possible that the change was merged but never deployed, and I would be deploying it live now [10:20:59] well, staging, in that case [10:21:17] can you check if it was rolled out to prod? [10:21:25] we can check SAL to be honest [10:23:30] I see 15:41 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply on Nov 13th [10:24:19] yea, prod diffs look clean. ok, i'll go ahead with staging. [10:24:28] grand [10:28:07] looking good, tests are still running. [10:28:49] I'd like to deploy a mw config patch as well, once the puppet change is in. Do you think I can just go ahead with that, or dhould I wait for a backport window? [10:29:59] it is up to you, backport windows are surely preferred [10:30:33] or we can wrap them both in the puppet window [10:30:51] ok, tests pass ,except the one that needs the mw config change. go head with the puppet change please! [10:33:39] re the config change: iirc releng prefers people who can deploy by themselves to do so outside the deployment windows... did that change? [10:34:31] I honestly do not know :) [10:34:52] but if you say so, lets keep going [10:35:13] what is your change ? just to add it in the window [10:35:37] (I will roll out now, just o have it in the calendar) [10:35:48] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1352093 [10:36:27] But i'm not available during the next window. Or the one after. It would have to wait until tomorrow. [10:36:43] no I will roll them out now [10:37:02] mediawiki infra is in 25' but we can start earlier [10:37:52] If we do it now I can test. If you'd do it later I can give you instructions for testing. [10:38:11] now is better, I wil merge puppet [10:38:13] I'll add it to the calendar for the next window [10:38:22] I am on it too dont worry [10:39:56] I'm just exited :) I have been wanting to do this for more than two years! [10:40:11] lol [10:42:10] let me know when the puppet change is through, so I can check that it didn't break anything [10:44:37] duesen: I will start running puppet on the european traffic servers [10:44:39] so you may test [10:51:45] effie: so... we broke something. [10:52:44] nothing ciritcal afaict [10:53:14] No need for an immediate rollback. But needs investigation, and if we can't figure it out, the puppet change would need to be reverted. [10:53:38] alright, how can I help? [10:53:43] because puppet is rolling out [10:53:54] being* rolled out [10:54:04] effect: things under /api/ are 404, even when they should not be. /apu/rest_v1 still works, so there is no need for immediate fire fighting [10:55:15] So, the idea is: ats routes /api/ to the gateway, the gateway routes /api/ to apache, apache routes /api/ to /w/rest.php. MediaWiki then parses the /api/ prefix (that's still missing). [10:56:21] There are different things under /api/: static files (/api/index.html), the old restbase apis (/api/rest_v1), and, in the future, all of mediawiki#s rest APIs (e.g. /api/va/page/Earth). The latter won't work until we have the config change. The first two were working before, ats routing directly to apache. [10:56:32] shall I ask for a traffic engineer to help out? [10:56:55] The poppet change made it so that this path would be routed via the gateway, which apparently doesn't work. By test on staging did work however. [10:57:47] It would be good to have someone with good ATS knowledge, yes. But it's possible that the probelm is in the gateway. I'll try to find a way to test from the deployment server using curl. the automated tests do that for staging, but not for prod. [10:59:01] are we able to know who returned the 404? ie rest-gw or apache? [10:59:16] you may see the 404s on the rest gw side maybe? [10:59:23] though mmm, no [10:59:48] wait [11:00:44] https://en.wikipedia.org/api/index.html [11:01:06] "server: envoy" [11:01:24] hold on. [11:01:28] solly me. I'm sorry. [11:01:33] yes :) [11:01:45] so it is rest-gw? [11:02:11] yes... because... i didn't actually deploy to prod... I canceled that after checking the diff... [11:02:16] I'm deploying to eqiad now. [11:02:18] haha [11:02:21] go for it [11:03:30] this is embarassing :D [11:04:10] oh daniel, if I were to count the times I have dome similar cockups, I wouldn't be here now [11:05:04] I'd be busking near the acropolis or something [11:05:36] :P [11:06:06] eqiad looks good, doing codfw [11:06:39] https://en.wikipedia.org/api/index.html fixed! [11:07:01] https://en.wikipedia.org/api/v1/page/Earth fixed too (still 404, but from mediawiki, not envoy) [11:07:07] *phew* [11:09:49] this is great [11:13:18] effie: yes! thank you very much! [11:13:28] can we go on with the config change? [11:13:32] you may backport your patch btw [11:13:42] I will move it to this windown [11:13:42] ok, i'll do that now then [11:13:47] (on wikitech) [11:13:53] ty! [11:16:59] I wonder if I will be able to test this on the debug hosts... I know that until revently, that didn't work for /w/rest.php. I wonder if explicit support would have to be added for this to work with the new /api route. [11:17:21] Clement did the /w/rest.php fix... I hope they also did /api... [11:18:37] let me know how I can help, I am missing some context here to be honest [11:20:37] the wikimedia-debug browser extension sends a header that should tell ATP (and the gateway? not sure how that works) to send traffic to debug servers. But apparently this has to be configued per route. I'll try and we'll see. It's no big deal if we can't test on debug hosts. [11:21:04] debug hosts = "testservers" [11:22:02] I think there was a task about it and it was sorted out, but please have a go [11:23:48] yes, it Just Works :) Confirmed on testservers. [11:23:53] grand [11:28:07] spiderpig deployment is still going [11:34:59] all done! thank you! 💃 [11:35:15] cheers daniel! [11:35:26] like and subscribe please [11:41:42] ⭐⭐⭐⭐⭐ [12:05:33] hahaha [12:45:08] duesen: XWD decision is completely offloaded to the rest-gateway for API paths