[00:08:39] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10API Platform: Migrate former Platform team service to GitLab - https://phabricator.wikimedia.org/T344739 (10Krinkle)
[00:29:19] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10AQS2.0, 10Thumbor: Migrate former Platform team service to GitLab - https://phabricator.wikimedia.org/T344739 (10Krinkle)
[00:30:51] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10AQS2.0, 10Thumbor: Migrate former Platform team service to GitLab - https://phabricator.wikimedia.org/T344739 (10Krinkle) Sorry, mistagged. While some of these were indeed in the "API Value Stream" of the former Platform Eng team, as I understand most/all of these w...
[00:45:34] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Fix CC BY-SA text on Phabricator - https://phabricator.wikimedia.org/T344965 (10Frostly)
[06:46:48] <wikibugs>	 (03CR) 10Hashar: "I have updated the Jenkins job ( https://integration.wikimedia.org/ci/job/wikidata-query-gui-build/ ) using from the root of integration/c" [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[06:49:24] <wikibugs>	 (03CR) 10Hashar: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[07:14:18] <wikibugs>	 (03CR) 10Hashar: "(I have manually disabled the job https://integration.wikimedia.org/ci/job/quibble-daily-EntitySchema-vendor-mysql-php74-docker/ since it " [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große)
[07:19:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Observability-Metrics: De-provision beta-specific Prometheus - https://phabricator.wikimedia.org/T344974 (10fgiunchedi)
[07:20:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Observability-Alerting, 10cloud-services-team, 10User-fgiunchedi: beta / deployment-prep alerts show up in production alertmanager - https://phabricator.wikimedia.org/T344582 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi >>! In T344582#9106603, @fgiun...
[07:21:58] <wikibugs>	 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T344975 (10JAllemandou)
[07:45:43] <wikibugs>	 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 20 - https://phabricator.wikimedia.org/T343826 (10hashar) 05Resolved→03Open We need the associated Jenkins jobs :)
[07:45:45] <wikibugs>	 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827 (10hashar)
[07:51:31] <wikibugs>	 (03CR) 10Noa wmde: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[08:34:00] <wikibugs>	 (03CR) 10Hashar: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[08:41:59] <wikibugs>	 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10HasanAkgun_WMDE) a:03HasanAkgun_WMDE
[08:42:05] <wikibugs>	 (03CR) 10Noa wmde: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[08:42:42] <wikibugs>	 (03CR) 10Noa wmde: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[08:45:13] <eoghan>	 Heads up, we'll be rebooting the phabricator hosts in 15 minutes. They should be unavailable for <10 minutes.
[08:46:36] <wikibugs>	 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech, 10Wikidata Dev Team (Sprint-∞): wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10HasanAkgun_WMDE)
[08:47:26] <wikibugs>	 10Project-Admins: Create project tag for Java-Scala-Standardization - https://phabricator.wikimedia.org/T344975 (10Peachey88)
[08:49:32] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Fix CC BY-SA text on Phabricator - https://phabricator.wikimedia.org/T344965 (10Lucas_Werkmeister_WMDE) I noticed this the other day as well… there is some “CC-BY-SA” [in Puppet](https://gerrit.wikimedia.org/g/operations/puppet/+/302a2bee98/modules/phabricator/data/fi...
[08:52:38] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Fix CC BY-SA text on Phabricator - https://phabricator.wikimedia.org/T344965 (10Aklapper) p:05Triage→03Low {T239355}/{T331673}; {T338440}
[08:53:34] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Remove a hyphen from "CC BY-SA" link text in Phabricator Footer  - https://phabricator.wikimedia.org/T344965 (10Aklapper)
[09:12:01] <eoghan>	 phabricator is back now
[09:37:27] <wikibugs>	 (03CR) 10Noa wmde: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[09:46:33] <wikibugs>	 10Phabricator: Exception: Map returned by "newPagingMapFromCursorObject()" in class "ManiphestTaskQuery" omits required key - https://phabricator.wikimedia.org/T344241 (10Aklapper) @thcipriani Do you have the search query URL by any chance? Steps to reproduce are always helpful for debugging :)
[09:51:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-LanguageSelector, 10Beta-Cluster-reproducible, 10JavaScript, and 3 others: Language selector broken: Uncaught TypeError: mw.uls.isNamed is not a function - https://phabricator.wikimedia.org/T344635 (10KartikMistry)
[10:04:48] <wikibugs>	 10Release-Engineering-Team, 10sre-alert-triage: Alert triage: overdue critical alert - https://phabricator.wikimedia.org/T342755 (10fgiunchedi) >>! In T342755#9118330, @thcipriani wrote: >>>! In T342755#9115945, @fgiunchedi wrote: >>>>! In T342755#9114368, @thcipriani wrote: >>> Hrm. We get an email from the s...
[10:17:11] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Physikerwelt)
[10:17:55] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services: Locking for gitlab backups - https://phabricator.wikimedia.org/T338332 (10eoghan) 05In progress→03Resolved This is deployed and seems to be working fine.  The one thing we haven't solved just yet is the fact that `rsync --delete` from the primary to the...
[10:28:39] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Physikerwelt) I see that there is already some outdated fork on gitlab https://gitlab.wikimedia.org/repos/releng/mathoid/. I think it would be great if we can use the latest ve...
[10:29:11] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Physikerwelt) p:05Triage→03Low
[10:33:22] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Physikerwelt)
[10:35:17] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Physikerwelt)
[10:56:43] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Physikerwelt) @thcipriani, maybe we can postpone this until extension-Math and extension-MathSearch are also moved to GitLab. At least it would be more convenient to configure...
[10:59:00] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Physikerwelt) Here is a list of related repositories https://gerrit.wikimedia.org/r/admin/repos/q/filter:math (mathoid-deploy is no longer needed).
[12:16:42] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Congratulations!" [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[12:17:53] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "You can still run the job against an open change made to `wikidata/query/gui` by changing the `ZUUL_REF` parameter from `master` to the re" [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[12:17:57] <wikibugs>	 (03Merged) 10jenkins-bot: wikidata-query-gui-build: run shell scripts in a container [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[12:40:30] <wikibugs>	 (03PS1) 10Hashar: Dockerfiles: add openssh-client to node test images [integration/config] - 10https://gerrit.wikimedia.org/r/952393 (https://phabricator.wikimedia.org/T328543)
[12:45:00] <wikibugs>	 (03PS2) 10Hashar: Dockerfiles: add openssh-client to node test images [integration/config] - 10https://gerrit.wikimedia.org/r/952393 (https://phabricator.wikimedia.org/T328543)
[12:45:55] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "The npm build and git push has been moved to a container and we require ssh. I will rebuild the image and switch the wikidata-query-gui-bu" [integration/config] - 10https://gerrit.wikimedia.org/r/952393 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[12:47:01] <wikibugs>	 (03Merged) 10jenkins-bot: Dockerfiles: add openssh-client to node test images [integration/config] - 10https://gerrit.wikimedia.org/r/952393 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[12:47:50] <wikibugs>	 (03PS1) 10Hashar: jjb: use image with ssh client for wikidata-query-gui-build [integration/config] - 10https://gerrit.wikimedia.org/r/952395 (https://phabricator.wikimedia.org/T328543)
[12:48:27] <wikibugs>	 (03CR) 10Hashar: "I will update the job once the new image has been build :)" [integration/config] - 10https://gerrit.wikimedia.org/r/952395 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[12:48:56] <hashar>	 !log Building Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/952393 # T328543
[12:48:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:48:59] <stashbot>	 T328543: [QB] [WDQS-GUI] Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543
[12:59:54] <_joe_>	 jnuche: can I ask you how is buildctl ran in CI by kokkuri? Do we run kokkuri inside a docker image, and we build the image inside the other image?
[13:04:41] <jnuche>	 _joe_: I think you got it right, we run Kokkuri inside of a prepared image (`docker-registry.wikimedia.org/repos/releng/kokkuri:v1.8.2` is the latest) and kokkuri invokes buildkit from there to build images
[13:07:22] <_joe_>	 jnuche: thanks, great
[13:07:48] <_joe_>	 context is - I'd like to add buildkit support to docker-pkg so we can run it in CI as well
[13:07:51] <_joe_>	 for base images
[13:11:25] <jnuche>	 then probably a similar mechanism is your best bet, the other option is having the tooling in the runner image, which is messier I believe
[13:11:48] <_joe_>	 so basically an image with the python code and buildkit
[13:12:23] <_joe_>	 I think I might steal some of your code :P
[13:12:30] <jnuche>	 in the case of kokkuri, yeah
[13:12:59] <_joe_>	 jnuche: yeah I'll do the same I think
[13:13:29] <_joe_>	 then we'll have that image run docker-pkg. I am just slightly worried about getting the jwt auth stuff right
[13:14:46] <jnuche>	 _joe_: this is all you should need: https://gitlab.wikimedia.org/repos/releng/kokkuri/-/blob/main/includes/kokkuri.yaml#L14
[13:15:13] <jnuche>	 gitlab CI requires the audience to be specified
[13:15:19] <jnuche>	 but we don't verify it
[13:16:05] <_joe_>	 oh great
[13:16:21] <jnuche>	 both the jwt-authorizer (and the layer cache if you want to use it, reggie) know how to handle the tokens generated by that config
[13:16:22] <_joe_>	 I should've asked you before the afternoon of the day before PTO
[13:16:33] <jnuche>	 hehehe
[13:16:38] <_joe_>	 I'll poke you again later in september I guess :D
[13:16:49] <jnuche>	 feel free to :)
[13:18:03] <_joe_>	 basically it would mean we can close the loop a bit on image building and concentrate it in a single place
[13:18:26] <_joe_>	 and I can then add later some callbacks when we publish images which we will need for tracking
[13:19:40] <jnuche>	 I'm not super knowledgeable on docker-pkg, but would it maybe make sense to merge it kokkuri at some point?
[13:19:52] <jnuche>	 *merge it with
[13:20:53] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10Jelto)
[13:21:11] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/952395 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[13:22:55] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: use image with ssh client for wikidata-query-gui-build [integration/config] - 10https://gerrit.wikimedia.org/r/952395 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[13:43:02] <wikibugs>	 (03CR) 10Noa wmde: wikidata-query-gui-build: run shell scripts in a container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/951066 (https://phabricator.wikimedia.org/T328543) (owner: 10Noa wmde)
[14:11:15] <wikibugs>	 (03PS1) 10Hashar: jjb: wikidata-query-gui-build push to git from Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/952451 (https://phabricator.wikimedia.org/T328543)
[14:46:42] <wikibugs>	 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for jdrewniak - https://phabricator.wikimedia.org/T344995 (10Jdrewniak)
[15:11:19] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "That unblocked the build https://integration.wikimedia.org/ci/job/wikidata-query-gui-build/87/console and the result is https://gerrit.wik" [integration/config] - 10https://gerrit.wikimedia.org/r/952451 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[15:12:24] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: wikidata-query-gui-build push to git from Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/952451 (https://phabricator.wikimedia.org/T328543) (owner: 10Hashar)
[15:13:52] <wikibugs>	 10Phabricator, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper)
[15:14:59] <wikibugs>	 10Phabricator, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper)
[15:22:12] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query Builder, 10Wikidata Query UI, and 6 others: [QB] [WDQS-GUI] Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10hashar) There were a few gotchas here and there due to how we run the CI jobs and some troubl...
[15:22:27] <hashar>	 I think that is the end of this week sprint for me :)
[15:22:36] <hashar>	 I am off for the week-end, be back on Monday!
[15:27:03] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥): Adopt gitlab-cloud-runner Digital Ocean based cluster as the default for untagged jobs - https://phabricator.wikimedia.org/T344874 (10xcollazo) (Original thread on Slack at https://wikimedia.slack.com/archives/C05H0JYT85V/p1692975749455829 )  TL;DR: This change br...
[15:31:32] <dancy>	 See ya Antoine!
[15:31:39] <Lucas_WMDE>	 enjoy the weekend!
[15:35:59] <Lucas_WMDE>	 whoa, zuul is completely empty right now
[15:36:33] <Lucas_WMDE>	 I don’t remember this usually happening even on friday afternoon ;)
[15:55:35] <wikibugs>	 10Project-Admins: Create project tag for Java-Scala-Standardization - https://phabricator.wikimedia.org/T344975 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #java-scala-standardization has been created: https://phabricator.wikimedia.org/project/view/6725/  (In case you need to edit...
[16:28:30] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10AlexisJazz)
[16:28:46] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10dduvall)
[16:30:20] <wikibugs>	 10Phabricator, 10Patch-For-Review, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/14  Remove some upstream...
[16:31:23] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper) a:03Aklapper https://gitlab.wikimedia.org/repos/phabricator/phabrica...
[16:33:14] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10taavi) How would this be different from #beta-cluster-reproducible?
[16:36:28] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10RhinosF1) @AlexisJazz: you can use train-blockers.toolforge.org to find the task for the current week. Most people probably wouldn't tag it correctly too in my opinion.
[16:37:38] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10AlexisJazz) >>! In T344999#9120584, @taavi wrote: > How would this be different from #beta-cluster-reproducible?  I had tagged T344635 with #beta-cluster-reproducible and it cl...
[16:41:47] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10dduvall) p:05Triage→03Medium a:03dduvall
[16:42:49] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10Aklapper) 05Open→03Declined > and it clearly didn't help. We won't fix either a social or a workflow problem by introducing competing workflows. :) Thus declining.  In my u...
[16:53:59] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10AlexisJazz) 05Declined→03Open >>! In T344999#9120614, @Aklapper wrote: >> and it clearly didn't help. > We won't fix either a social or a workflow problem by introducing co...
[16:57:15] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10Aklapper) 05Open→03Stalled p:05Triage→03Low > Unlike you, I'm not getting paid so telling me how I should work, especially when it's not particularly intuitive, is unli...
[17:19:31] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10AlexisJazz) >>! In T344999#9120647, @Aklapper wrote: >> Unlike you, I'm not getting paid so telling me how I should work, especially when it's not particularly intuitive, is un...
[17:31:24] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10CodeReviewBot) dduvall opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-ru...
[18:22:42] <wikibugs>	 10Project-Admins: Create a beta cluster equivalent for the Wikimedia-production-error tag - https://phabricator.wikimedia.org/T344999 (10Aklapper) > so why didn't YOU tag T344635 as a train blocker? As you see, playing the blame game is easy, but it doesn't help anyone.  Either nobody realized that it should hav...
[18:36:27] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T343725 (10dduvall) 05Open→03Resolved
[19:10:46] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T343726 (10thcipriani) p:05Triage→03Medium a:03jeena
[19:38:37] <wikibugs>	 10Scap, 10Data-Engineering, 10Data-Platform-SRE: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 (10VirginiaPoundstone)
[20:19:32] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[20:21:26] <James_F>	 (Sorry, my fault.)
[20:24:53] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Jdforrester-WMF) >>! In T344747#9119945, @Physikerwelt wrote: > @thcipriani, maybe we can postpone this until extension-Math and extension-MathSearch are also moved to GitLab....
[20:28:51] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10
[20:39:14] <inflatador>	 brennen I could be making a lot of assumptions here, but I'm seeing an alert for Zuul gearman in alerts and we're not getting responses on https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/937535/ , any chance those are related?
[20:39:32] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[20:54:15] <inflatador>	 nm, looks like just busy as opposed to broken
[20:56:45] <RhinosF1>	 inflatador: you can use integration.wikimedia.org/Zuul to see what's running
[20:56:59] <RhinosF1>	 https://integration.wikimedia.org/zuul/
[20:57:04] <RhinosF1>	 Lowercase Z
[20:57:21] <RhinosF1>	 It's busy ye
[20:57:35] <inflatador>	 RhinosF1 ACK, I was staring blankly at it...but James-F set me straight in #sure
[20:57:39] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[20:57:41] <inflatador>	 errr...#sre that is
[20:58:03] <RhinosF1>	 inflatador: ye I just saw
[21:07:49] <thcipriani>	 yeah, it's sure busy. stuff is running so I doubt it's broken.
[21:07:53] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[21:15:33] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10
[23:34:45] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Priority Backlog 📥): Archive the "wikimedia/production" repo - https://phabricator.wikimedia.org/T344763 (10thcipriani) >>! In T344763#9113913, @Jdforrester-WMF wrote: > I've [[https://gerrit.wikimedia.org/r/admin/repos/wikimedia/production,g...