[00:41:32] 06serviceops, 10MW-on-K8s: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429 (10RLazarus) 03NEW [00:47:09] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10270816 (10RLazarus) >>! In T341553#10268045, @Michael wrote: > Recording here that I'm noticing myself still running one-off scripts on the maint-hosts because, as I understand... [08:49:01] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10271255 (10Ammarpad) >>! In T378385#10269970, @Scott_French wrote: > Thanks, @kostajh. > > FWIW, it looks like `Kategorie:Wikipedia:Seite,_die_JsonConfig_verwen... [09:44:11] hey folks! [09:44:20] I have a patch to migrate aux-k8s to containerd https://gerrit.wikimedia.org/r/c/operations/puppet/+/1083776 [09:44:32] our dear Janis is not available otherwise I'd pester him :D [10:46:13] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10271594 (10aborrero) I noticed this alert today: {F57655150} I'm not sure if it is related to the work in this... [10:55:35] 06serviceops, 10MW-on-K8s: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10271653 (10Clement_Goubert) Ranked by order of preference: - 3 seems like the "cleanest", a spin on this could be to deploy a couple of config files specific to `mw-script` running w... [12:19:33] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Add helper script functionality to our php images - https://phabricator.wikimedia.org/T377958#10272041 (10Clement_Goubert) Helper scripts deployed and available in the mediawiki containers: `lang=bash root@deploy2002:~# kubectl -n mw-debug exec mw-debug.eqiad.pi... [12:23:22] elukey: +1'd because I think the blast radius isn't huge, doesn't mean I'm actually certain it's correct :D [12:24:05] thanks! [12:25:48] 06serviceops, 10MW-on-K8s: Allow using helper scripts inside of mwscript-k8s - https://phabricator.wikimedia.org/T378479 (10Clement_Goubert) 03NEW [12:27:30] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Add helper script functionality to our php images - https://phabricator.wikimedia.org/T377958#10272098 (10Clement_Goubert) [12:42:10] !log Killed dead and stacked import-wikitech.sh processes on wikitech-static - T374114 [12:58:05] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10272227 (10Clement_Goubert) The import process seems to stall out after a few hundred pages, ~~I'll try to upgrad... [13:24:52] 06serviceops, 10MW-on-K8s: Allow using helper scripts inside of mwscript-k8s - https://phabricator.wikimedia.org/T378479#10272539 (10Urbanecm_WMF) Related note: I sometimes need to run a script on an arbitrary set of wikis (I have the set of wikis in a txt file, rather as a committed dblist). Currently, there... [15:12:27] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10273089 (10Clement_Goubert) This is {T348503} again, reopening this task. [16:18:19] hnowlan: o/ I have a question related to how Thumbor was migrated to k8s in the past, from metal to wikikube.. IIRC you were able to add to the discovery endpoint, initially only served by bare metal thumbors, thumbor pods on wikikube. Am I recalling it correctly? [16:18:51] I am asking since we'd probably need to do something like that with kartotherian (or better, it would be nice if we were able to do it) [16:21:40] 06serviceops, 10MW-on-K8s: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10273612 (10thcipriani) I'm listed as approver for `restricted` and `deployment`. When I evaluate requests for additions to those groups, I aim for [[https://en.wikipedia.org/wiki/Prin... [16:25:53] 06serviceops, 10Wikimedia-Site-requests, 10WMF-General-or-Unknown, 13Patch-For-Review: Setup missing.php layer redirects for wikipedia hosting the other projects too - https://phabricator.wikimedia.org/T376923#10273636 (10Pppery) I am not going to redirect Russian language wikinewses like udm.wikinews.org... [16:29:47] elukey: yep! It started off as me manually adding kube workers to the pool of hosts, for testing and capacity purposes etc [16:30:09] and then over time adding more and more of them, ultimately removing the old hosts and the finally cleaning up the dedicated pools and using the kubesvc [16:30:53] 06serviceops, 10Wikimedia-Site-requests, 10WMF-General-or-Unknown, 13Patch-For-Review: Setup missing.php layer redirects for wikipedia hosting the other projects too - https://phabricator.wikimedia.org/T376923#10273669 (10Pppery) I am not going to redirect udm.wikinews.org -> ru.wikinews.org unless someone... [16:35:08] hnowlan: ahh I see https://gerrit.wikimedia.org/r/c/operations/puppet/+/951545 [16:35:21] okok so now I have a clearer idea, looks very promising thanks! [16:39:52] np, lemme know if you need anything more on it or review. it was all pretty painless except for some hiccups with cleanup [16:40:43] the only thing that it could be annoying right now is to pool a lot of k8s wikikube workers to the kartotherian discovery endpoint [16:41:02] but maybe we can pin the pods to specific hosts (temporarily) and add only those [16:42:02] basically this https://gerrit.wikimedia.org/r/c/operations/puppet/+/951546/2/conftool-data/node/codfw.yaml nowadays would be a way longer list :D [17:13:35] so, if there are concerns about the number of nodes to add to the `maps` pool, technically it wouldn't need to be all of them (i.e., connections hitting the nodeport service will random-LB across pods anyway) [17:14:27] this is a very good point! [17:14:40] I guess the one tricky thing here is the port targeted by the backing LVS service (443) [17:15:36] maybe that can change in the baremetal config first, so that it will be easier to add k8s nodes [17:16:05] I'll study something [17:16:08] thanks! [17:17:53] sounds good - there _might_ be some interesting ipvsadm wrangling to make that happen, but probably no insurmountable :) [17:18:00] *not [17:18:27] if we can avoid those I'd be happier :D [17:53:13] 06serviceops, 13Patch-For-Review: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752#10274113 (10Reedy) >>! In T373752#10136087, @Scott_French wrote: > @Reedy - Got it, thanks for the context! I'll look into how complicated this might be on 7.4 in light of how... [18:58:20] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10274455 (10Scott_French) While I don't anticipate any issues (and indeed have not observed any so far), I'm going to let syntaxhighlight soak for a bit before merging https://gerrit.wikimedia.org... [20:20:10] 06serviceops, 10Wikimedia-Site-requests, 10WMF-General-or-Unknown, 13Patch-For-Review: Setup missing.php layer redirects for wikipedia hosting the other projects too - https://phabricator.wikimedia.org/T376923#10274655 (10Pppery) Another thing that could in theory be done here if I was going to go for sema... [20:55:55] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10274767 (10cscott) I don't think they are related. That same tracking category appeared on every wiki, nothing explains why it would have cause a load spike for...