[07:04:08] 06serviceops, 10Parsoid (Tracking): Cleanup parsoid-php service - https://phabricator.wikimedia.org/T359387#10034452 (10akosiaris) 05Open→03Resolved a:03akosiaris Node reimaged, pooled with weight 10 and uncordoned. I 'll happily resolve this, the legacy parsoid cluster is no more! [07:42:27] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad: decommission deploy1002 - https://phabricator.wikimedia.org/T371283#10034513 (10akosiaris) [07:44:00] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad: decommission deploy1002 - https://phabricator.wikimedia.org/T371283#10034520 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by akosiaris@cumin1002 for hosts: `deploy1002.eqiad.wmnet` - deploy1002.eqiad.wmnet (**PASS**) - Down... [07:56:29] 06serviceops, 06collaboration-services, 06SRE, 10Release-Engineering-Team (Radar): replace production buster deployment servers - https://phabricator.wikimedia.org/T364656#10034567 (10akosiaris) 05Open→03Resolved a:03akosiaris deploy1003 has been tracked in T364417, deploy2002 reimaging as bullse... [08:56:53] FYI heads up, I'm going ahead with switching k8s logs to dedicated kafka topics in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1057819 functionally nothing will change [09:17:56] godog: thanks <3 [09:23:00] yw claime <3 [10:14:10] 06serviceops, 06SRE, 07Epic, 13Patch-For-Review: Phase out cergen for ServiceOps services - https://phabricator.wikimedia.org/T360636#10034762 (10Clement_Goubert) [10:21:23] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: MVP: Privately server wiktech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10034765 (10jijiki) >>! In T371537#10033727, @bd808 wrote: > The config thing that most needs to be changed to use the multiversion images is `/etc/mediawiki/WikitechPr... [10:54:49] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10034841 (10jijiki) [10:55:28] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10034843 (10jijiki) [10:59:52] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10034846 (10akosiaris) [11:29:29] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034931 (10jijiki) [11:29:40] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10034936 (10jijiki) [11:30:09] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034934 (10jijiki) [11:30:41] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10034938 (10jijiki) [11:43:15] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034963 (10Bugreporter) [11:45:30] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10034971 (10jijiki) [11:49:54] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034980 (10jijiki) [11:59:16] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034990 (10jijiki) →14Duplicate dup:03T359820 [11:59:17] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034995 (10jijiki) [11:59:21] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10034996 (10jijiki) [12:54:02] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10035257 (10jijiki) [12:54:07] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10035258 (10jijiki) [12:59:16] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10035291 (10jijiki) [13:36:12] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Memcached: Migrate wikitech to main memcached - https://phabricator.wikimedia.org/T371608 (10jijiki) 03NEW [13:38:48] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10035449 (10jijiki) a:03Ladsgroup [13:49:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10035491 (10Jclark-ctr) wikikube-worker1260 3183 #. 1 wikikube-worker1261 3182 #. 0 wikikube-worker1262 3184 #. 2 wikikube-worker1263... [14:05:11] akosiaris hnowlan should i go ahead and deploy this patch? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1059048 [14:07:21] nemo-yiannis: go for it. For the most part we shouldn't see much change until another patch is deployed, correct? [14:07:35] the restbase patch was deployed yesterday [14:07:49] so summary should be aware of if-unmodified-since [14:09:41] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10035577 (10jijiki) [14:09:42] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Apache: Include Wikitech in mw-on-k8s' virtual hosts - https://phabricator.wikimedia.org/T371360#10035578 (10jijiki) [14:11:31] ack [14:15:40] nemo-yiannis: go for it [14:18:06] ugh, i just found an issue on restbase while manually testing the headers, sigh [14:18:16] i think its easy to fix though [14:29:22] narrator: it wasn't. [14:29:27] :P [14:31:56] lol, no it is: https://github.com/wikimedia/restbase/pull/1349 [14:35:21] \o/ [15:09:14] should i try both restbase and changeprop deployments now ? [15:12:37] no objection from me [15:14:32] ok [15:15:57] it was requests to mw-api-int that increased last time right? [15:16:17] yeah [15:16:36] * claime watches graphs [15:16:57] but now (hopefully) we should see a reduction [15:23:48] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10035804 (10jijiki) a:03jijiki [15:36:00] ok restbase is done [15:36:09] doing changeprop now [15:52:02] changeprop deployment is done as well [15:52:35] cool, watching [15:55:24] i didn't see any meaningful reduction in summary requests to backend :/ [15:56:33] 06serviceops: Migrate poolcounter hosts to bullseye - https://phabricator.wikimedia.org/T332015#10035970 (10Dzahn) [15:58:43] 06serviceops: Migrate poolcounter hosts to bullseye - https://phabricator.wikimedia.org/T332015#10035982 (10Dzahn) T370458 is asking for help getting the same thing done in beta. Both will need the poolcounter-prometheus-exporter package for bullseye/bookworm. [15:59:16] 06serviceops: Migrate poolcounter hosts to bullseye - https://phabricator.wikimedia.org/T332015#10035989 (10akosiaris) > poolcounter-prometheus-exporter will need to be packaged for bullseye No need for that. It's a golang, I 've copied it to bookworm-wikimedia. poolcounter itself is also on bookworm already. [15:59:59] 06serviceops: Migrate poolcounter hosts to bullseye - https://phabricator.wikimedia.org/T332015#10035991 (10Dzahn) Cool, i'll pass on the good news! [16:06:16] hnowlan: next thing we can try is to keep the logic for the if-unmodified-since and point page/summary to MW resource changes and not parsoid to see if we have the same bump in requests like last time [16:06:31] but i am running out of ideas atm [16:19:12] but with this change we shouldn't see much change in pattern right? we're still preserving the old i-u-s behaviour in addition to this without changing the parsoid routing [16:19:19] or am I misunderstanding the changes that have been made up to this point [16:29:30] true, but we do have *some* (few) pregeneration rules that follow mediawiki resource change [16:29:40] could be the case that there is not enough traffic to show up [16:30:00] i saw very few 412 for page summary which means that the rule works [16:30:09] and zero for page/mobile-html which also makes sense [16:32:14] but do those existing rules go via restbase? [16:33:00] I'd be up for trying to change page/summary but probably not on a thursday evening ;) [16:33:12] yeah for sure, maybe next week [16:33:41] the rules at the moment go via restbase yes [16:33:53] i will keep an eye, but i didn't see any issues so far [16:35:09] yeah I don't see anything alarming atm - might be worth mentioning in #wikimedia-sre that the changes are in place just in case anything explodes overnight [19:37:31] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission deploy1002 - https://phabricator.wikimedia.org/T371283#10036634 (10VRiley-WMF) 05Open→03Resolved [19:37:41] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission deploy1002 - https://phabricator.wikimedia.org/T371283#10036637 (10VRiley-WMF) This unit has been decommissioned [20:45:06] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T371045#10036734 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF This unit has been relabeled as requested. Thanks! [21:29:53] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10036924 (10dr0ptp4kt) To confirm understanding, did we have a leaning on whether the containerized version... [23:52:10] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037213 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bull... [23:52:16] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1251.eqiad.wmnet with OS bull... [23:52:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037215 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1252.eqiad.wmnet with OS bull... [23:52:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1253.eqiad.wmnet with OS bull... [23:52:48] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037217 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1254.eqiad.wmnet with OS bull... [23:53:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037220 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1257.eqiad.wmnet with OS bull... [23:55:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037225 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1258.eqiad.wmnet with OS bull... [23:55:54] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037226 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1259.eqiad.wmnet with OS bull... [23:55:54] 06serviceops, 13Patch-For-Review: Support warmup for local caches in mw-on-k8s - https://phabricator.wikimedia.org/T369921#10037227 (10Scott_French) With the cache_warmup class relocated, I think the near-term work is done. There are two TODOs related to fully removing the script etc. from the maintenance... [23:55:58] 06serviceops, 13Patch-For-Review: Support warmup for local caches in mw-on-k8s - https://phabricator.wikimedia.org/T369921#10037228 (10Scott_French) 05Open→03Resolved [23:56:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037230 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1256.eqiad.wmnet with OS bull... [23:56:06] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037231 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1255.eqiad.wmnet with OS bull...