[07:10:19] 06serviceops, 10Scap: Reimage deploy2002 as bullseye - https://phabricator.wikimedia.org/T371282#10026280 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host deploy2002.codfw.wmnet with OS bullseye [07:53:58] hello folks! [07:54:07] o/ [07:55:08] Would it be possible to depool wikikube-worker1240 to re-run the provision cookbook? More details in https://phabricator.wikimedia.org/T371132, but the TL;DR is that due to a supposedly no-op refactor by yours truly some bios settings haven't been applied [07:55:27] running the provision cookbook may trigger a reboot, this is why I am asking about the depool [07:55:28] it's not pooled [07:55:30] it's a new host [07:55:44] so, do whatever you want [07:55:52] they aren't even imaged fwiw [07:56:02] PERFECT [07:56:06] <3 :) [07:56:24] I was planning to image it in ~30minutes for the very first time, I 'll hold, let me know when you are done [08:11:23] 06serviceops, 10Scap: Reimage deploy2002 as bullseye - https://phabricator.wikimedia.org/T371282#10026434 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host deploy2002.codfw.wmnet with OS bullseye completed: - deploy2002 (**PASS**) - Downtimed on Icinga/Alertm... [08:12:40] elukey: btw, you got a disabled puppet on kafka-main2001 (16H already). Nothing just in case you forgot about it [08:21:52] 06serviceops, 10Scap: Reimage deploy2002 as bullseye - https://phabricator.wikimedia.org/T371282#10026445 (10akosiaris) 05Open→03Resolved Reimage done, /home synced, keyholder armed. [09:30:55] akosiaris: o/ back sorry I was at the dentist.. re-enabling, I think I disabled it when there was the issue with corrupted data [10:01:53] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10026777 (10jijiki) [10:03:07] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10026770 (10jijiki) 05Open→03In progress p:05Medium→03High [10:06:15] 06serviceops, 06SRE, 10wikitech.wikimedia.org: Install php-ldap on all MW appservers - https://phabricator.wikimedia.org/T237889#10026798 (10jijiki) [10:06:37] 06serviceops, 06Data-Persistence, 06SRE, 07Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907#10026813 (10jijiki) [10:11:21] akosiaris: green light for wikikube-worker1240 [11:23:47] Thanks [11:35:18] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10026982 (10akosiaris) [11:38:55] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Traffic: Direct wikitech.wikimedia.org to mw-on-k8s - https://phabricator.wikimedia.org/T371358 (10jijiki) 03NEW [11:40:26] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359 (10jijiki) 03NEW [11:41:29] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Apache: Include Wikitech in mw-on-k8s' virtual hosts - https://phabricator.wikimedia.org/T371360 (10jijiki) 03NEW [12:54:38] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374 (10jijiki) 03NEW [12:54:41] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10027442 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1240.eqiad.wmnet with OS bull... [12:55:57] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10027449 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1240.eqiad.wmnet with OS bullseye... [12:56:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10027451 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1240.eqiad.wmnet with OS bull... [12:59:28] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10027469 (10jijiki) [13:04:59] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378 (10jijiki) 03NEW [13:39:39] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10027682 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1240.eqiad.wmnet with OS bullseye... [13:44:44] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: Cleanup parsoid-php service - https://phabricator.wikimedia.org/T359387#10027684 (10akosiaris) [13:58:16] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T371260#10027726 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [14:17:26] hello again folks [14:17:39] I'd need to run the provision cookbook again for some wikikube nodes, list in https://phabricator.wikimedia.org/T371132 [14:18:09] In theory no need to a reboot, in practice I'd like to depool first just in case [14:20:45] elukey: drain + pooled=inactive, do you want me to do it? [14:21:46] claime: nono just wanted to know if it was ok, to avoid messing up with ongoing maintenance etc.. [14:23:54] elukey: hmmm be careful, the mw nodes you have have in that list have been renamed [14:24:23] ah snap [14:24:40] never a joy :D [14:24:48] see https://phabricator.wikimedia.org/T370672 https://phabricator.wikimedia.org/T371260 [14:29:17] <3 [14:44:18] claime: the provision cookbook is likely rebooting nodes, I'll drain them before applying it [14:45:22] elukey: ack, remember to set them inactive in conftool as well in case a scap deployment is needed while you're doing it [14:46:51] * elukey takes notes [15:10:14] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Beta-Cluster-Infrastructure, 06SRE, 10Wikidata: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#10028314 (10Krinkle) @Urbanecm_WMF Do you mind uploading it to Gerrit under 06serviceops, 10Beta-Cluster-Infrastructure, 06SRE, 10Wikidata, 10wmde-wikidata-tech: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#10028317 (10Krinkle) [15:34:17] 06serviceops, 10Beta-Cluster-Infrastructure, 06SRE, 10Wikidata, 10wmde-wikidata-tech: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#10028409 (10Urbanecm_WMF) >>! In T125976#10028314, @Krinkle wrote: > @Urbanecm_WMF Do you mind uploading it to Gerrit under... [16:26:14] 06serviceops, 13Patch-For-Review: Support warmup for local caches in mw-on-k8s - https://phabricator.wikimedia.org/T369921#10028589 (10Scott_French) [16:26:15] 06serviceops, 07Datacenter-Switchover: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10028590 (10Scott_French) [16:43:53] claime: kudos. resolving a "wiki rename" task from 2011 is crazy. Those seemed like they will never be resolved. [16:44:16] you know it's old when a task is imported from Bugzilla :) [16:56:12] hello, heads up! i'll be deploying an apache rewrite rule for mediawiki.org in just a few minutes! https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240730T1600 [16:56:32] best of luck! [16:56:33] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1052791 [16:56:54] apache rewrite rules in our env always gimme the chills [16:57:45] i have chills [16:58:02] to confirm, I will merge that puppet patch, run puppet on deploy host, then do [16:58:03] scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False [16:58:07] https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#The_scap_way [16:58:09] does that sound right? [17:01:31] ottomata: yes, that's right. I'm not 100% sure how the diffs will render for the configmaps (i.e., they might be rather large). just flagging since `--k8s-confirm-diff` may present you with a lot of text :) [17:01:39] okay thank you [17:01:50] while i'm here: unrelated questions: what is the recommended way to access noc.wikimedia.org from within wmf prod network? [17:02:09] is it [17:02:10] e.g. [17:02:11] curl -s -H 'Host: noc.wikimedia.org' https://mw-misc.discovery.wmnet:30443/conf/dblists/open.dblist [17:02:11] ? [17:02:19] mw-misc 30443 ? [17:05:33] swfrench-wmf: okay, I see the change I want in mediawiki-pinkunicorn-httpd-sites-config. but there are also a lot of diffs for unrelated stuff [17:05:35] e.g. [17:05:35] mw-misc, mediawiki-main-envoy-config-volume, ConfigMap (v1) has changed: [17:05:53] lots of removals of thigns for parsoid-php? [17:06:16] also - # Network egress to parsoid-php is being removed [17:06:24] ottomata: hmm ... interesting. let me take a look at the diffs [17:06:38] okay, shall I enter 'N' for Continue with sync? [17:06:48] sure, yeah that sounds good [17:07:04] done [17:07:12] haha, btw the prompt is: [17:07:12] Continue with sync? [y/N]: [17:07:19] but 'N' does not work, it just reprompts [17:07:20] 'n' works tho [17:07:21] :p [17:07:34] https://www.irccloud.com/pastebin/L2p75yvo/ [17:07:42] lol [17:09:39] alright, I see: the parsoid diffs are from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1006900 which removed the envoy listener [17:10:23] k! shall I proceed? [17:10:28] or, you are welcome to if you are in there [17:12:13] ottomata: you're good to go, IMO [17:12:33] I was just scanning through the changes a.kosiaris made earlier today [17:12:46] okay! proceeding [17:19:39] deploy done. testing rewrite... [17:22:31] awesome - fingers crossed :) [17:22:39] 06serviceops, 10Wikimedia-Apache-configuration, 10Wikimedia-Site-requests: Temporarily redirect sgs.wikipedia.org to bat-smg.wikipedia.org until bat-smg->sgs move can be done - https://phabricator.wikimedia.org/T204830#10028896 (10Pppery) [17:25:28] i need to access mediawiki.org bypassing varnish to test. I'm trying [17:25:28] curl -v 'Host: www.mediawiki.org' 'https://mw-api-int-ro.discovery.wmnet:4446 [17:25:30] but getting [17:25:40]

This domain points to a Wikimedia Foundation server, but is not configured on this server.

[17:27:00] swfrench-wmf: any advice? [17:27:59] OH [17:28:03] wait, am I missing -H [17:28:04] ummmm [17:28:17] was just about to ask :) [17:28:18] DOH sorry for the noise, operator error [17:28:26] no worries! [17:28:46] awww maaaannnn [17:28:52] The requested path (/beacon/event) was not inside the REST API base path (/w/rest.php)" [17:29:39] i think my rule needs to modify the path and kind proxy the requets maybe? [17:29:39] hmmm [17:29:48] okay back to the most recent drawing board... [17:31:21] ah, sorry to hear that =/ pulling up the patch to take a look [17:31:42] swfrench-wmf: for related previous questions: [17:31:42] what is the recommended way to access noc.wikimedia.org from within wmf prod network? [17:31:42] curl -s -H 'Host: noc.wikimedia.org' https://mw-misc.discovery.wmnet:30443/conf/dblists/open.dblist [17:31:42] ? [17:32:59] ah, right! so, could you give a bit more context on the intended use case? (e.g., accessing noc.wikimedia.org should just work from hosts where one might right, say, a one-off script that reads this) [17:33:29] s/right/run/ [17:33:38] swfrench-wmf: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/774#note_97651 [17:33:50] basically: the job here needs to discover the list of full list of wiki dbs [17:37:29] ah, got it - so this is a bit more than a one-off [17:38:03] ya, regular job, will be used by dumps 2.0 [17:38:11] ottomata: how often does it run? [17:38:44] so, +1 to not going out and "coming in the front door" via noc.wm.o, and agreed that targeting an appropriate discovery address is preferable [17:39:02] but it feels like there should be a better way to access this information [17:39:57] cdanis: once a day [17:40:16] sounds like a one-off to me ;) [17:40:29] automated one off? :) [17:41:03] so one request at job start, with one job start per day [17:41:19] yup, it might be run manually sometimes for backfilling, etc. [17:42:19] IMO, while it would be nice to find some other way to provide y'all with this information, this seems like a fine solution given your use case / frequency [17:42:28] ^ agree [17:42:41] please do add a clear UA to the request, though, so we can track this down if needed [17:43:31] and swfrench-wmf is accesing via CDN noc.wm.org okay? [17:43:43] or should we do mw-misc? [17:43:45] fwiw, you could also pull that from https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/dblists/open.dblist or https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/dblists/open.dblist [17:44:16] mutante: yes, Xabriel had that at first, but there is no normal way to get text out of ^. gitiles only gives you base64 encoded [17:44:29] ah, *nod* [17:44:43] which we could do, but it is I htink worse than asking noc.wm.org [17:44:51] and more difficult to change just by changing the url [17:45:01] or clone the mediawiki-config repo somewhere ? [17:45:44] ya I suggested that too. adds more complexity thouugh: means we have to bring in puppet, and make sure the thign is cloned and updated. Also airflow is moving to k8s, so we'd have to deal with cloning a repo in k8s... [17:46:05] gotcha, yea. just wanted to list options. I see you already considered them [17:46:11] thank you! :) [17:47:37] it's possible gitiles might be replaced by a different tool for repo browsing, but that won't help right now of course [17:48:19] ottomata: I would tend to agree that a once-a-day call to noc.wm.o is probably fine [17:48:23] ottomata: this is 'raw file" though, btw: https://phab.wmfusercontent.org/file/data/zmsxkqi7x4reem7akony/PHID-FILE-chkr3n3m7vetvhem7hnt/open.dblist [17:48:33] well! that would work too! [17:48:40] which is better? haha [17:48:54] mutante: is that a commit pinned one? or just from master? [17:49:53] that should be just from master [17:50:12] the "Raw File" button from https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/dblists/open.dblist [17:50:39] mutante: do you think the file from phab/git is better? or going through a deployed service noc.wm.org? [17:51:14] swfrench-wmf: thank you. back to my rewrite problem... [17:51:30] any tips for testing rewrite rules with mw? i'm finding that i'm just guessing here. Maybe I need PT pass thru? [17:52:48] Oo, or mayb ejust P for proxy? [17:56:03] ottomata: i don't see a big difference. assuming either they are generated from repo or uploaded to repo right after being generated. So I guess whatever is easier for you. well... when using noc it stays in the internal network. And maybe a slightly higher risk that the phab repo browsing could be replace by something in the future. [17:56:39] that being said, dunno if noc is supposed to stay as it is forever or we would like to replace that too with links to a repo [17:56:45] i have a slight prefernce for noc. accessing the repo is okay, but seems a bit more of an 'internal concern' than noc [17:57:01] ottomata: PT sounds plausible if there are other rewrite rules that are necessary for the "right hand side" of your rewrite to work as expected, but I would very much defer to _j.oe_ and others more familiar with the details of apache config :) [17:57:08] e.g. if iit goes somewhere else, noc could set up a redirect [17:57:31] i guess...i'll try to see if i can get mod_rewrite in my local mw docker... [17:57:40] now to learn about how apache is setup there.. [17:57:44] ottomata: just to confirm, there's nothing "broken" about the rewrite rule being live as it is right now, right? (i.e., this doesn't need reverted) [17:58:05] no, not broken. it can't be reached until we remove the varnish handling of the url its trying to match anyway [17:58:21] awesome, thanks :) [17:58:26] thank you for your help! [18:27:23] I think I want to do [18:27:24] https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Testing_config [18:40:35] ottomata: that's probably the best we've got for testing something like this out, yeah [18:40:44] okay thank you. [18:40:52] good thing i have sudo ! ;) [18:42:34] :) [18:42:44] TIL --connect-to ! [18:42:52] much easier than setting Host header! [18:49:24] I'm trying to use [P] which uses mod proxy [18:49:24] getting [18:49:30] AH01144: No protocol handler was valid for the URL /beacon/event (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule. [18:50:18] do I need mod ssl enabled? [19:04:25] 06serviceops, 06DC-Ops, 10ops-eqiad: Install (2) 960GB SSDs each in kafka-main10[06-10] - https://phabricator.wikimedia.org/T371422 (10RobH) 03NEW p:05Triage→03Medium [19:04:26] 06serviceops, 06DC-Ops, 10ops-codfw: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423 (10RobH) 03NEW p:05Triage→03Medium [19:13:42] ottomata: sorry, missed this before. I don't think we have mod_proxy loaded, and also can't think of an existing use case for P. maybe start with PT instead? [19:14:29] PT has same problem: [19:14:45] MW Rest API is failing because it sees the incoming uri path as /beacon/event, and it explicitly fails [19:14:59] if it does not start with e.g. /w/rest.php [19:15:35] i think mod_proxy is enabled [19:15:38] https://www.irccloud.com/pastebin/wekNgNzc/ [19:15:40] but [19:15:55] I thikn some other handler / module it needs, isn't [19:16:51] I need to: [19:16:51] - accept the request from /beacon/event [19:16:51] - and route the request to /w/rest.php/eventlogging/v0/beacon [19:21:24] swfrench-wmf: can I test enabling modules? [19:21:48] i think mabye we need https://httpd.apache.org/docs/current/mod/mod_proxy_http.html [19:24:34] or maybe [19:24:35] https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslproxyengine [19:24:57] ottomata: alas, I don't have a great answer for that ... in general, I would assume that doing so more broadly would be involved to vet appropriately [19:25:08] yes agree [19:30:37] added comment [19:30:37] https://phabricator.wikimedia.org/T353817#10029516 [19:30:51] _joe_: ^ when you have moment would appreciate some advice. thank you! [19:39:00] <_joe_> ottomata: I gave the correct advice on june 8th [19:39:03] <_joe_> https://phabricator.wikimedia.org/T353817#9959609 [19:39:56] <_joe_> but if we have to solve your current problem, the issue is how the rest api reads its path [19:40:15] <_joe_> it would fail with any form of url rewriting that's not proxying properly [19:41:19] yeah... what sucks is I couldn't (didnt' try hard enough to) test the rewrite locally before doing all this rest api work. [19:41:20] sigh. [19:41:50] a pro of doing it as API endpoint was the code was all in EL extension. [19:41:57] buuut i guess we go back to the old way [19:42:28] <_joe_> ottomata: but your proxy rules are not going to work for sure, yes [19:42:38] _joe_: unless you think otherwise, I don't think it is worth enabling more apache modules just for this. [19:42:53] so: back to the mw-config /w php file way [19:42:56] <_joe_> ottomata: there is no way we do, but also it's not needed [19:43:02] oh? [19:43:38] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10029621 (10RobH) [19:43:55] <_joe_> ottomata: give me tomorrow to play with it a bit, do you happen to know what _SERVER variable does the rest api use to understand the path? [19:44:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Install (2) 960GB SSDs each in kafka-main10[06-10] - https://phabricator.wikimedia.org/T371422#10029625 (10RobH) [19:44:54] <_joe_> (the correct solution to this is to build a url handler that isn't this rigid and where we can add internal rewrites, all in php, instead of forcing us to do apache config sorceries, but that's a lost battel) [19:45:52] I was just looking at this to try to understand what's happening, and _think_ it's getting eventually to `WebRequest::getGlobalRequestURL` which would in turn default to `$_SERVER['REQUEST_URI']` ... which I think means it's using something that will never see the rewritten path? [19:46:12] (looking...) [19:47:10] <_joe_> swfrench-wmf: well the problem is that this deviates from what MediaWiki always did heh :) [19:47:37] <_joe_> but yes, then the only option is a proxypass I think, which is tbh out of the question [19:47:44] not 100% sure of course :) [19:48:27] <_joe_> ottomata: you could also make your file in /w just check the uri is what you want, mangle the needed _SERVER variable, include your rest endpoint :) [19:48:54] nasty :) [19:49:43] if i have to put a php file in /w and rewrite to it, i might as well put logic in it that just calls the proper functions, rather than going through MW REST framework [19:50:46] <_joe_> it's not nasty tob :) [19:51:01] true [19:51:03] its all nasty [19:51:11] <_joe_> it's the first version of the entry point we'll use eventually :D [19:51:21] <_joe_> and we could forever blame you for how unelegant it is! [19:52:00] hahah /w/ottorouter.php becomes a standard [19:55:47] <_joe_> ++ [19:56:00] <_joe_> that's how the best stuff we have was born [20:55:04] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10029921 (10bd808) The "very special" bits are I think just the legacy setup from being on it's own strange pair of hosts detached from all other MediaWiki deployments. I can't think o... [22:28:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1241.eqiad.wmnet with OS bull... [22:28:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030178 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1243.eqiad.wmnet with OS bull... [22:28:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030179 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1242.eqiad.wmnet with OS bull... [22:32:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1245.eqiad.wmnet with OS bull... [22:48:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030204 (10Jclark-ctr) [22:56:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030221 (10Jclark-ctr) [23:06:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030228 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1244.eqiad.wmnet with OS bull... [23:06:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1242.eqiad.wmnet with OS bullseye... [23:06:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030230 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1246.eqiad.wmnet with OS bull... [23:09:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030233 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1243.eqiad.wmnet with OS bullseye... [23:09:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030235 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1247.eqiad.wmnet with OS bull... [23:12:58] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030243 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1245.eqiad.wmnet with OS bullseye... [23:15:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030250 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1249.eqiad.wmnet with OS bull... [23:15:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030251 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1241.eqiad.wmnet with OS bullseye... [23:19:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030258 (10Jclark-ctr) [23:26:22] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030268 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1248.eqiad.wmnet with OS bull... [23:44:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030281 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1244.eqiad.wmnet with OS bullseye... [23:45:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030283 (10Jclark-ctr) [23:46:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030285 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1246.eqiad.wmnet with OS bullseye... [23:46:41] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030286 (10Jclark-ctr) [23:49:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030291 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1247.eqiad.wmnet with OS bullseye... [23:53:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030296 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1249.eqiad.wmnet with OS bullseye... [23:53:25] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10030297 (10Jclark-ctr)