[07:41:50] Hi folks! I want to deploy an opdate to the rest-gateway in a couple of hours, four patches starting at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1260763/3. I'll polish and rebase them now, then they should be ready to go. What time would be convenient for deployment? Who could be around to rescue me in a pinch? So far, these deployments have mostly gone very smooth, and when there were issues we caught them on [07:41:50] staging. But I'd still feel better if there was some around, just in case :) [08:21:37] duesen: I will be around, I would suggest to either use the backport window (after normal backport is done) or the infra window right after [08:21:55] would that work? [08:23:47] oh sorry I am wrong, I thought the UTC morning backport window was earlier [08:24:23] duesen: would the MediaWiki infrastructure (UTC mid-day) 10:00–11:00 UTC window work? [08:30:27] effie: yes that should work, thanks! [08:31:03] cheers, if it is not too much trouble, do you mind adding it in the dev cal? [08:52:37] effie: hm... which calender? can you give me a link? [08:53:31] https://wikitech.wikimedia.org/wiki/Deployments :) [08:53:43] oh on wiki! sure. [08:53:48] hehe [08:57:51] done [08:58:47] <3 [09:49:13] I don't think adding it to the calendar is necessary in this case, nobody else is going to be deploying rest-gateway at the same time, right? [09:54:22] I would say it is good practice to keep changes visible there too, along with the bot on -operations announcing that something is going on [10:00:36] the ping from the bot is also a nice touch [10:06:12] effie: [10:06:33] ...Raine : i'm here now, starting to hit +2 if it's fine with you [10:07:03] the plan is to deploy the first four patches in one go, then deploy the final one (centralauthtoken support) [10:07:26] sgtm [10:07:34] ugh, Okta just logged me out of everything [10:07:59] -_- [10:11:02] Raine: we are back to using two data centers properly, yes? [10:11:13] and i should deploy to codfw first? [10:11:17] we're still currently drained out of eqiad [10:11:28] ah ok [10:11:38] (if that's problematic, i think we're probably good to undrain, i just need to do a quick capacity calculation) [10:12:59] bjensen: wait, I see 10k req/sec on the gateway in equiad, but basically nothing in codfw. so... we are drained out of... codfw? [10:13:12] sorry, yes [10:13:12] i don't think it's a problem, no. [10:13:17] switchover has me dizzy :) [10:13:22] :P [10:14:31] But I should still be deploying from codfw? Or should I deploy form eqiad now? [10:15:54] Right now it doesn't matter [10:16:33] Once we're in both, you should be starting with eqiad because that will be the secondary, read traffic only DC for the next half a year [10:16:51] So might as well start with eqiad already [10:17:37] ok. wikitech says that 1003 is the primary deployment server, so i'll do it from there. [10:18:53] Oh, in terms of which deployment server to use, yes, always use the current primary one [10:19:07] I thought you were asking which DC to deploy to first [10:20:01] despite the name, deployment.eqiad.wmnet will always point to the active deployment server in the active dc [10:20:01] missed the chart bump... will do it now... at least i noticed before applying [10:20:20] Raine: both :) [10:41:47] applying to staging... [10:44:41] running make check... [10:45:22] ok, everything passes. [10:46:09] Raine: i'll start with eqiad, because then i can verify the effect of the patch. no good "testing" on a dc that has no traffic. [10:46:20] ack [10:58:27] ok, first deployment looks sane, applying to codfw. I'll hit +2 on the last patch [11:06:57] the centralauthtoken patch is deployed to staging, running make check [11:10:56] one test flaked out, running again [11:15:46] different test flaked out this time. looks ok [11:16:46] applying to eqiad [11:27:18] ok, the deployment is successful. the feature isn't quite working as expected, but didn't make things worse. I'll apply to codfw and follow up later. [11:40:38] 👍