[00:11:40] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351#10907152 (10Krinkle) //Bolding tagging on the assumption that CI-infra is not monitored as an inbox or workboard. Feel free to... [00:48:41] (03update) 10brennen: SpiderPig log view [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/767 (https://phabricator.wikimedia.org/T391005) [00:56:46] (03update) 10brennen: SpiderPig log view [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/767 (https://phabricator.wikimedia.org/T391005) [07:34:33] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10907852 (10Esanders) [07:41:02] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10907862 (10A_smart_kitten) [07:41:13] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10907864 (10A_smart_kitten) [08:25:03] (03PS1) 10Bartosz Wójtowicz: inference-services: Add missing CI job configuration for pre-commit-check. [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) [09:43:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:43:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732 (10wmcs-alerts) 03NEW [09:58:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:00:49] (03CR) 10Ilias Sarantopoulos: [C:03+1] "Sorry, I totally missed that during the review!" [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [10:03:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:12:33] (03CR) 10Hashar: [C:03+2] "Fun fact: I should have caught it myself when I approved the other change 😂" [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [10:14:33] (03Merged) 10jenkins-bot: inference-services: Add missing CI job configuration for pre-commit-check. [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [10:23:33] (03open) 10phuedx: Add "experiments" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [10:23:42] (03update) 10phuedx: Add "experiments" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [10:32:53] bit of a long queue for CI jobs at the moment (but as I'm typing this it seems to be clearing so..) [10:33:44] * TheresNoTime thinks the "reporting an issue so it immediately resolves itself" is a valid strategy for fixing stuff here [10:37:34] TheresNoTime: it's a valid fix everywhere [10:37:43] :D [10:38:16] TheresNoTime: did it at work too. Rang IT cause something was stuck being virus scanned and it released as soon as they picked up. [10:41:47] "have you tried turning it off and on again?" [10:42:27] (03CR) 10Hashar: [C:03+2] "Deployed! Looks like I forgot to do it immediately cause I am juggling with a bunch of other tasks :/" [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [10:43:39] TheresNoTime: yes there is a series of changes made to Wikibase [10:43:56] at least php7.4 jobs got removed, so that is less problematic than it used to be [10:44:13] I was just being impatient! :) [10:58:45] 10GitLab (Infrastructure), 10Cloud-Services, 06collaboration-services: Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739 (10Jelto) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedi... [10:59:46] 10GitLab (Infrastructure), 06collaboration-services, 10VPS-Projects: Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10908647 (10Jelto) [11:00:56] 10GitLab (Infrastructure), 06cloud-services-team, 10Cloud-VPS, 06collaboration-services: Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10908653 (10taavi) [11:02:01] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10908655 (10Jelto) p:05Triage→03Medium a:03Jelto I created a new instance `gitlab-1001` in `devtools`. But I can not attache the volume with the ba... [11:11:29] TheresNoTime: Being inpatient because you have no time? [11:23:26] 10Beta-Cluster-Infrastructure: Add basic MediaWiki/web site up alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695#10908711 (10taavi) >>! In T315695#10905945, @bd808 wrote: > I don't want to see you get stuck being the only human who knows how the bits fit together. I'm afraid I dug myself... [11:27:46] (03update) 10phuedx: Add "experiments" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [11:27:54] (03update) 10phuedx: Add "experiments" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [11:30:03] (03CR) 10Bartosz Wójtowicz: "Thank you for all the help and reviews Antoine <3 The CI stage is now triggered correctly in inference-services repo" [integration/config] - 10https://gerrit.wikimedia.org/r/1156254 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [11:32:58] * Lucas_WMDE fortunately checked https://bash.toolforge.org/search before !bash’ing that TheresNoTime quote [11:52:09] (03open) 10jelto: add example how to build Dockerfile images [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/126 [11:54:13] (03update) 10jelto: add example how to build Dockerfile images [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/126 [11:54:19] (03update) 10jelto: add example how to build Dockerfile images [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/126 [12:00:20] GitLab needs a short maintenance restart in around 30 minutes [12:24:23] (03CR) 10Hashar: "I am going to remove the `php-ast` image by inlining the building into each of the php images (php81, php82, php83, php84): If0abfacaf54cd" [integration/config] - 10https://gerrit.wikimedia.org/r/1154873 (owner: 10Jforrester) [12:50:11] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10908963 (10Urbanecm_WMF) [12:52:03] 10Continuous-Integration-Config, 10phan, 10phan-taint-check-plugin, 13Patch-For-Review, and 2 others: Figure out why php-ast sometimes loses MAGIC_* flags in PHP 8.3 - https://phabricator.wikimedia.org/T396312#10908966 (10hashar) a:03hashar [12:59:28] oh my [13:00:04] the six images `node??-test-browser-php??-composer` have the php `ast` extension [13:03:44] (03open) 10jelto: gitlab-runner: bump image version to alpine-v17.10.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/475 (https://phabricator.wikimedia.org/T396701) [13:05:54] ## END: O, my kingdom for multiple inheritance! [13:05:54] * [13:05:55] yeah [13:05:57] :/ [13:19:30] (03PS7) 10Hashar: dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) [13:20:44] (03CR) 10CI reject: [V:04-1] dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [13:30:35] (03PS1) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [13:31:55] (03CR) 10CI reject: [V:04-1] dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [13:32:49] (03update) 10phuedx: Add "experiments" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [13:54:59] (03update) 10phuedx: Add "xlab" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [13:55:16] (03update) 10phuedx: Add "xlab" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) [13:59:42] (03PS8) 10Hashar: dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) [13:59:42] (03PS2) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [14:00:52] (03CR) 10CI reject: [V:04-1] dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:00:56] (03CR) 10CI reject: [V:04-1] dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:00:58] pff [14:05:36] (03PS9) 10Hashar: dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) [14:05:36] (03PS3) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [14:06:44] (03PS4) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [14:06:50] (03CR) 10CI reject: [V:04-1] dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:08:02] (03CR) 10CI reject: [V:04-1] dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:10:30] (03PS5) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [14:12:49] (03PS1) 10Hashar: dockerfiles: remove php-ast image [integration/config] - 10https://gerrit.wikimedia.org/r/1156363 (https://phabricator.wikimedia.org/T396312) [14:17:38] (03CR) 10Hashar: "This is introducing a `php-scratch` image that has a script to compile php-ast and a PHP script to test the compiled extension." [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:22:35] (03CR) 10Jforrester: "This fails locally for me with:" [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:23:01] what did I broke :b [14:25:53] hashar: Everything, of course. [14:27:43] I forgot some `--from=php-scratch` in COPY statements [14:27:46] Ah. [14:27:53] Want me to fix? [14:27:57] I did :) [14:28:01] I am rebuilding locally [14:28:04] Ack. [14:28:13] I did not caught it cause the images were already built in my local store :/ [14:28:19] * James_F nods. [14:28:41] I almost want to patch docker-pkg to have a --test-run command that makes the images with a timestamp version. [14:29:13] there is `--snapshot` [14:29:25] but I think it rebuilds all the parents as well [14:29:49] $ docker-pkg --snapshot [14:29:49] usage: docker-pkg [-h] [-c CONFIGFILE] [--debug | --info] {build,prune,update} ... [14:30:03] Nothing in the help. Of course. [14:30:16] yeah :b [14:30:26] patch welcome I guess [14:30:26] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/docker-pkg/+/refs/heads/master/docker_pkg/cli.py#77 [14:30:33] but I don't know exactly what are the diffs between the two [14:30:37] or what they are supposed to do [14:31:48] (03CR) 10Hashar: "I forgot to update some `COPY` statements to use `--from=php-scratch`." [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:31:56] (03PS10) 10Hashar: dockerfiles: php: build ast with same php version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) [14:31:56] (03PS6) 10Hashar: dockerfiles: node-test-brower-php-composer: build ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) [14:31:57] (03PS2) 10Hashar: dockerfiles: remove php-ast image [integration/config] - 10https://gerrit.wikimedia.org/r/1156363 (https://phabricator.wikimedia.org/T396312) [14:32:27] James_F: I also sent a patch to enhance logging: https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/1156315 [14:32:31] which I should probably split [14:32:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of zuul VPS project - https://phabricator.wikimedia.org/T396540#10909397 (10fnegri) 05Open→03Resolved Project created! Please double check that the permissions and quotas ar... [14:32:39] and regardless that needs a proper commit message [14:32:49] Indeed. And maybe a task. :-P [14:33:05] dhinus: success! thank you for the project creation! [14:33:20] yw! [14:36:00] (03PS11) 10Jforrester: Docker: [php*] Build php-ast with the exact same PHP version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:45:31] (03CR) 10Jforrester: [C:03+2] Docker: [php*] Build php-ast with the exact same PHP version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:47:03] hashar: For the node-test-php images, I was thinking about whether it's instead possible to just copy /etc/php and /bin/php or whatever from php81/etc. images, rather than re-build everything from Sury? This'd make things "simpler" to manage at our end, if less clean. Thoughts? [14:47:58] (03Merged) 10jenkins-bot: Docker: [php*] Build php-ast with the exact same PHP version [integration/config] - 10https://gerrit.wikimedia.org/r/1155742 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [14:50:35] !log Docker: [php*] Build php-ast with the exact same PHP version, for T396312 [14:50:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:50:38] T396312: Figure out why php-ast sometimes loses MAGIC_* flags in PHP 8.3 - https://phabricator.wikimedia.org/T396312 [14:53:10] James_F: re node-test-php images: nop, cause installing the packages do install extra dependencies etc [14:53:20] Yeah. :-( [14:53:36] Why can't apt just create foo.app packages like on a Mac? ;-) [14:53:49] I have used `vimdiff dockerfiles/node??-test-browser-php??-composer/Dockerfile.template to edit them [14:53:58] My deepest sympathies. [14:54:01] and really they only differ by the php version and the node ones [14:54:11] Yeah, maybe we should parameterise them somehow? [14:54:42] I think that is doable by having one master copy `node-test-browser-php-composer` [14:54:52] which has the Dockerfile.template / control [14:55:11] It was originally a one-off special case. [14:55:13] Alas. [14:55:19] then create more specific dirs that do symlink the Dockerfile.template [14:55:42] and the template could maybe somehow COPY an environment file that list the version of node and php to use [14:55:45] eg: NODE_VERSION=22 [14:55:50] PHP_VERSION=8.2 [14:56:01] Yeah. [14:56:12] We should also try to lock down the PHP_VERSION in the php\d\d images. [14:56:15] and then the master Dockerfile would have something such as: {{ "php-$PHP_VERSION node-$NODE_VERSION" | apt_install }} [14:56:23] But as sury doesn't release multiples, that's harder. [14:56:25] but that is just my fringe theory [14:56:34] the main issue is that we have multi inheritance [14:56:43] or polymorphism (I think that is the term in OOO) [14:56:52] But at least it'd stop us shipping php90:9.0.0-s3 which is actually php90:9.0.4. [14:56:56] Yeah. [14:56:58] and Docker/overlayfs only support a tree [14:57:09] OR [14:57:16] To quote myself: [14:57:17] "## START: O, my kingdom for multiple inheritance!" [14:57:30] we teach docker-pkg to be able to generate several images out of a single template [14:57:36] yeah [14:57:36] :) [14:57:51] so I just copy pasted [14:57:53] docker-pkg --tokyo-drift [14:57:56] :b [14:58:12] anyway the image should be fine, there is at least a php script to validate ast is good [14:58:23] I am off, I have a sailling competition this evening [14:58:33] I'll build them/update jobs tommorrow if you don't beat me to it [14:58:34] :) [14:59:21] We need to cascade the images. [14:59:31] php81 etc. aren't ever used directly. [14:59:56] Clearly you didn't properly use docker-pkg, but at least we have working base images now. [15:00:00] yeah I did not do the cascade intentionally: I wanted to keep the change small [15:00:33] so either the change can be amended after running docker-pkg update (but maybe that will also update the node-test-php-composer images which are then updated iagain in a child change) [15:01:32] also we can probably the number of php images [15:01:34] I'll do this first. [15:01:40] (03PS7) 10Jforrester: Docker: [node-test-brower-php*-composer] Build php-ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [15:01:50] php, composer-test, composer, composer-package, php-compile [15:01:58] Yuuup. [15:02:00] they can probably all be aggregated in a single php image [15:02:03] that has the dev tools [15:02:04] composer [15:02:07] and ast :) [15:02:14] And composer-scratch. [15:02:27] (03CR) 10Jforrester: [C:03+2] Docker: [node-test-brower-php*-composer] Build php-ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [15:02:31] <3 [15:02:33] thanks!! [15:02:47] I am off [15:02:57] (looks like Phan will be fixed) [15:03:02] Yeah. [15:03:43] (03Abandoned) 10Jforrester: Docker: [php-ast] Drop PHP 7.4 and PHP 8.0 builds, no longer supported [integration/config] - 10https://gerrit.wikimedia.org/r/1154873 (owner: 10Jforrester) [15:03:51] (03Merged) 10jenkins-bot: Docker: [node-test-brower-php*-composer] Build php-ast inline [integration/config] - 10https://gerrit.wikimedia.org/r/1156350 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [15:04:07] !log Docker: [node-test-brower-php*-composer] Build php-ast inline, for T396312 [15:04:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:04:13] T396312: Figure out why php-ast sometimes loses MAGIC_* flags in PHP 8.3 - https://phabricator.wikimedia.org/T396312 [15:07:48] (03open) 10dancy: .gitlab-ci.yml: Bump to docker-registry.wikimedia.org/python3-devel:0.0.4-20250608 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/476 [15:07:50] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services: Investigate Zuul worker nodes for post-merge/trusted jobs - https://phabricator.wikimedia.org/T396655#10909575 (10thcipriani) Per discussions between ServiceOps and @Jelto the Aux cluster is... [15:07:51] (03update) 10dancy: .gitlab-ci.yml: Bump to docker-registry.wikimedia.org/python3-devel:0.0.4-20250608 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/476 [15:08:21] (03update) 10dancy: .gitlab-ci.yml: Bump to docker-registry.wikimedia.org/python3-devel:0.0.4-20250608 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/476 [15:08:38] hashar: Do we want to switch the quibble images over to doing the same? Currently they're getting the special new version of php-ast from sury… [15:09:18] (03merge) 10dancy: .gitlab-ci.yml: Bump to docker-registry.wikimedia.org/python3-devel:0.0.4-20250608 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/476 [15:09:42] (03update) 10dancy: gitlab-runner: bump image version to alpine-v17.10.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/475 (https://phabricator.wikimedia.org/T396701) (owner: 10jelto) [15:11:07] (03PS1) 10Jforrester: Docker: Cascade uses of php* with new php-ast inline build [integration/config] - 10https://gerrit.wikimedia.org/r/1156383 (https://phabricator.wikimedia.org/T396312) [15:11:43] (03merge) 10dancy: gitlab-runner: bump image version to alpine-v17.10.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/475 (https://phabricator.wikimedia.org/T396701) (owner: 10jelto) [15:11:54] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10909596 (10brennen) [15:16:49] 10GitLab (Infrastructure), 06collaboration-services, 07SecTeam Discussion, 07Security, 05Vuln-VulnComponent: GitLab Security Release: 18.0.2, 17.11.4, 17.10.8 - https://phabricator.wikimedia.org/T396701#10909666 (10sbassett) [15:22:12] TheresNoTime: Sorry for accidentally rollback'ing your reply to me. I clicked thank, and the re-flow moved my click target and so ended up on the wrong link. "Naturally" thanking someone needs a confirmation but rolling back doesn't. Sigh. [15:23:16] !log Upgraded gitlab-cloud-runners to v17.10.2 (T396701) [15:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:23:18] T396701: GitLab Security Release: 18.0.2, 17.11.4, 17.10.8 - https://phabricator.wikimedia.org/T396701 [15:23:50] James_F: ah no worries :D [15:28:24] (03open) 10dancy: blubber.yaml: Bump to docker-registry.wikimedia.org/bullseye:20250608 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/127 [15:28:27] (03update) 10dancy: blubber.yaml: Bump to docker-registry.wikimedia.org/bullseye:20250608 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/127 [15:31:10] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10909804 (10jijiki) Current status: * POC working on wikikube-worker2100.codfw.wmnet, tested with h... [15:33:01] (03merge) 10dancy: blubber.yaml: Bump to docker-registry.wikimedia.org/bullseye:20250608 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/127 [15:33:59] (03update) 10dancy: image.py: Pin dependencies to avoid click bool/envvar regression [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/124 (owner: 10dduvall) [15:35:22] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767 (10jijiki) 03NEW [15:36:47] (03merge) 10dancy: image.py: Pin dependencies to avoid click bool/envvar regression [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/124 (owner: 10dduvall) [15:36:49] (03update) 10dancy: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 (owner: 10dduvall) [15:37:22] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10909858 (10jijiki) 05Open→03In progress p:05Triage→03Medium [15:38:26] (03update) 10dancy: add example how to build Dockerfile images [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/126 (owner: 10jelto) [15:43:34] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services: Investigate Zuul worker nodes for post-merge/trusted jobs - https://phabricator.wikimedia.org/T396655#10909883 (10thcipriani) Clarification about available drivers for nodepool to use in prod... [16:14:35] (03PS1) 10Jforrester: jjb: Switch node-php image users over to inline-ast-built ones [integration/config] - 10https://gerrit.wikimedia.org/r/1156394 [16:22:02] (03PS2) 10Jforrester: jjb: Switch node-php image users over to inline-ast-built ones [integration/config] - 10https://gerrit.wikimedia.org/r/1156394 [16:22:21] (03CR) 10Jforrester: [C:03+2] jjb: Switch node-php image users over to inline-ast-built ones [integration/config] - 10https://gerrit.wikimedia.org/r/1156394 (owner: 10Jforrester) [16:22:49] (03CR) 10Jforrester: [C:03+2] Docker: Cascade uses of php* with new php-ast inline build [integration/config] - 10https://gerrit.wikimedia.org/r/1156383 (https://phabricator.wikimedia.org/T396312) (owner: 10Jforrester) [16:23:36] (03PS3) 10Jforrester: Docker: Drop php-ast image, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1156363 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [16:23:39] (03PS4) 10Jforrester: Docker: Drop php-ast image, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1156363 (https://phabricator.wikimedia.org/T396312) (owner: 10Hashar) [16:23:48] (03Merged) 10jenkins-bot: jjb: Switch node-php image users over to inline-ast-built ones [integration/config] - 10https://gerrit.wikimedia.org/r/1156394 (owner: 10Jforrester) [16:24:03] (03Merged) 10jenkins-bot: Docker: Cascade uses of php* with new php-ast inline build [integration/config] - 10https://gerrit.wikimedia.org/r/1156383 (https://phabricator.wikimedia.org/T396312) (owner: 10Jforrester) [16:24:43] !log Docker: Cascade uses of php* with new php-ast inline build, for T396312 [16:24:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:24:45] T396312: Figure out why php-ast sometimes loses MAGIC_* flags in PHP 8.3 - https://phabricator.wikimedia.org/T396312 [16:35:14] (03open) 10dduvall: image.py: Prevent dockerfile forwarding to syntax line frontends [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/128 [16:36:20] (03update) 10dduvall: image.py: Prevent dockerfile forwarding to syntax line frontends [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/128 [16:36:24] (03update) 10dduvall: image.py: Prevent dockerfile forwarding to syntax line frontends [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/128 [16:41:35] (03approved) 10dancy: image.py: Prevent dockerfile forwarding to syntax line frontends [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/128 (owner: 10dduvall) [16:42:56] (03open) 10dduvall: ci: Pass BUILDKIT_SYNTAX to avoid dockerfile frontend forwarding [repos/releng/buildx] - 10https://gitlab.wikimedia.org/repos/releng/buildx/-/merge_requests/2 [16:45:49] 10GitLab (Infrastructure), 06collaboration-services, 07SecTeam Discussion, 07Security, 05Vuln-VulnComponent: GitLab Security Release: 18.0.2, 17.11.4, 17.10.8 - https://phabricator.wikimedia.org/T396701#10910322 (10Dzahn) [16:48:01] (03merge) 10dduvall: ci: Pass BUILDKIT_SYNTAX to avoid dockerfile frontend forwarding [repos/releng/buildx] - 10https://gitlab.wikimedia.org/repos/releng/buildx/-/merge_requests/2 [16:50:18] (03merge) 10dduvall: image.py: Prevent dockerfile forwarding to syntax line frontends [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/128 [17:08:15] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [17:08:17] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [17:08:18] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [17:12:32] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [17:12:32] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [17:12:34] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [18:03:32] 10Continuous-Integration-Config, 07Epic, 07PHP 8.4 support: Make PHP 8.4 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T386108#10910580 (10Krinkle) [18:10:08] 10Continuous-Integration-Config, 10phan, 10phan-taint-check-plugin, 13Patch-For-Review, and 2 others: Figure out why php-ast sometimes loses MAGIC_* flags in PHP 8.3 - https://phabricator.wikimedia.org/T396312#10910615 (10Krinkle) [18:10:25] (03update) 10brennen: SpiderPig log view [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/767 (https://phabricator.wikimedia.org/T391005) [18:31:56] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10910713 (10brennen) [18:54:26] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Write release announcement for 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390699#10910765 (10MSantos) 05Open→03In progress [19:11:20] (03approved) 10dancy: SpiderPig log view [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/767 (https://phabricator.wikimedia.org/T391005) (owner: 10brennen) [19:16:53] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:16:57] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:16:58] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:30:47] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Write release announcement for 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390699#10910833 (10MSantos) 05In progress→03Resolved [19:35:57] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:35:57] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:36:09] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:51:27] (03open) 10dancy: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 [19:51:29] (03update) 10dancy: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 [19:56:02] (03update) 10dancy: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 [19:57:47] (03update) 10dancy: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 [20:11:33] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732#10911004 (10bd808) `counterexample $ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Inf... [20:12:30] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732#10911005 (10bd808) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1155609 [20:15:14] !log Added `profile::memcached::firewall_srange: ~` to deployment-memc Puppet prefix (T396732) [20:15:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:15:16] T396732: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732 [20:16:50] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732#10911008 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ba4200721c2e21433babeafa355e469816f7fd0f%5E%21/#F0 ` diff --git a... [20:18:47] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732#10911010 (10bd808) 05Open→03Resolved p:05Triage→03Medium a:03bd808 [20:28:29] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:28:46] !log `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes (T396748) [20:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:29:05] wtf memc11 I just fixed you [20:32:13] [20:32:29] I think this is alertmamanger being wacky [20:32:43] [20:33:00] computers were a mistake. this is known [20:36:09] I double checked and `sudo -i puppet agent -tv` is clean on all 3 memc nodes. I just acked the alert in alertmanager to make it hush. I assume it will figure things out in a bit. [20:38:29] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:38:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396800 (10wmcs-alerts) 03NEW [20:39:18] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396732#10911058 (10bd808) [20:39:19] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc12 on project deployment-prep - https://phabricator.wikimedia.org/T396800#10911060 (10bd808) →14Duplicate dup:03T396732 [20:41:42] !log `sudo service varnish-frontend restart` on deployment-cache-text08 to pick up blocked_nets changes (T394881) [20:41:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:44:19] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10911092 (10bd808) [20:48:09] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [20:48:11] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [20:48:13] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:40:11] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:40:12] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:40:14] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:45:25] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:45:25] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:45:26] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:46:20] 10Beta-Cluster-Infrastructure: Add basic MediaWiki/web site up alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695#10911213 (10bd808) Dumping some notes to my future self here. Following hints from https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VP... [22:10:50] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:10:51] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:10:53] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:23:04] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:23:06] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:23:09] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:32:12] 10Beta-Cluster-Infrastructure: Request creation of new Beta Cluster wiki - https://phabricator.wikimedia.org/T396809 (10Nhu_Gay_Me) 03NEW [22:34:49] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:34:51] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:34:52] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:37:58] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:37:59] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:38:02] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:38:05] sorry for the noise :D [22:55:06] 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T392175#10911379 (10brennen) Nearing end of US workday and still blocked on T396790. We'll call this parked overnight and roll forward Friday once a... [22:55:45] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:55:46] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:55:51] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:58:12] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:58:12] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:58:15] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:18:36] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:18:37] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:18:40] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:23:38] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:23:39] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:23:41] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:32:21] 10Beta-Cluster-Infrastructure: Request creation of a cebwiki in Beta Cluster - https://phabricator.wikimedia.org/T396809#10911443 (10bd808) [23:34:29] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:34:30] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:34:32] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:42:23] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:42:24] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:42:27] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:48:50] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:48:50] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:48:53] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:52:46] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:52:48] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:52:50] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:56:27] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:56:28] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:56:30] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [23:56:32] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125