[00:42:08] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11581562 (10Dzahn) >>! In T405119#11564356, @dduvall wrote: > @Dzahn After debugging for a bit with @thcipriani I finally just l...
[07:01:38] <wikibugs>	 (03PS1) 10Jforrester: Zuul: […/OATHAuth] Add dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236605
[07:01:38] <wikibugs>	 (03PS1) 10Jforrester: Zuul: […/OATHAuth] Add phan dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236606
[07:02:03] <wikibugs>	 (03CR) 10Jforrester: "Per the recent breakage, it sounds like this is needed." [integration/config] - 10https://gerrit.wikimedia.org/r/1236605 (owner: 10Jforrester)
[07:02:22] <wikibugs>	 (03CR) 10Jforrester: "It's not clear to me whether this is needed." [integration/config] - 10https://gerrit.wikimedia.org/r/1236606 (owner: 10Jforrester)
[08:49:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-REST-API, 05MW-1.46-notes (1.46.0-wmf.14; 2026-02-03), 06MW-Interfaces-Team (MWI-Sprint-26 (2026-01-27 to 2026-02-10)): REST API broken on beta sites - https://phabricator.wikimedia.org/T415771#11582177 (10hashar)
[08:49:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Page Content Service, 06Content-Transform-Team (Work In Progress), 13Patch-For-Review: PCS/summary returns errors for beta requests - https://phabricator.wikimedia.org/T415877#11582180 (10hashar) →14Duplicate dup:03T415771
[09:30:00] <kevinbazira>	 o/ the Jenkins CI is failing on the inference services pipeline pre-commit check: https://phabricator.wikimedia.org/P88633
[09:30:00] <kevinbazira>	 what could be causing this?
[09:53:03] <hashar>	 WorkflowScript: Loading libraries failed
[09:53:16] <hashar>	 kevinbazira: that is a java update. I gotta restart all of the jenkins instances
[09:53:30] <hashar>	 it only affects pipelines jobs though
[09:53:42] <hashar>	 I think it is something such as Jenkins running under Java version n-1
[09:53:55] <kevinbazira>	 great. thanks hashar :)
[09:54:06] <hashar>	 but compiles/loads/whatever integration/pipelinelib.git on demand using whatever is the new Java version currently on the box
[09:54:14] <hashar>	 so there is some serialization issue between the two
[09:54:16] <hashar>	 and that explodes
[09:54:21] <hashar>	 DO NOT QUOTE ME ON THIS
[09:54:22] <hashar>	 :b
[09:59:25] <kevinbazira>	 np. will wait for your confirmation when CI has been fixed :)
[09:59:41] <hashar>	 kevinbazira: sure!
[09:59:47] <hashar>	 I am waiting for some jobs to complete
[10:03:54] <hashar>	 !log Restarted Jenkins on releases2003.codfw.wmnet
[10:03:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:13:58] <hashar>	 kevinbazira: I have restarted everything!
[10:14:57] <kevinbazira>	 merci beaucoup ... let me check
[10:15:14] <wmf-insecte>	 Yippee, build fixed!
[10:15:14] <wmf-insecte>	 Project beta-code-update-eqiad build #586121: 09FIXED in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/586121/
[10:17:54] <kevinbazira>	 hashar: CI job has succeeded. thanks a lot! <3
[10:29:54] <hashar>	 kevinbazira: you are welcome, thank you for the ping! :]
[10:30:11] <hashar>	 I should have restarted Jenkins yesterday night when I was made aware java got upgraded
[10:30:19] <hashar>	 :)
[10:32:07] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11582481 (10hashar)
[10:38:22] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11582511 (10hashar) I have rolled back the train due to an issue with OAuth on metawiki: {T416456}
[10:55:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 778115 integration-agent-docker-1042 (/: 36%, /srv: 100%, /var/lib/docker: 30%): OFFLINE due to disk space
[11:00:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 778116 integration-agent-docker-1042 (/: 36%, /srv: 59%, /var/lib/docker: 30%): RECOVERY disk space OK
[11:30:14] * hashar lunches
[12:52:17] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Petition] Add CLDR dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236274 (owner: 10Fomafix)
[12:53:53] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236274 (owner: 10Fomafix)
[12:54:51] <James_F>	 !log Zuul: [mediawiki/extensions/Petition] Add CLDR dependency
[12:54:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:11:51] <wikibugs>	 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471 (10Mvolz) 03NEW
[13:22:40] <wikibugs>	 06Project-Admins: Create project tag for Wishlist-FY25-26 - https://phabricator.wikimedia.org/T415913#11583199 (10KSiebert) @Aklapper I had not answered some of your previous questions:   > How could I currently find [some of] these wishes that you'd like to tag in this ticket? We find them using https://meta.wi...
[13:30:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 778146 integration-agent-docker-1043 (/: 36%, /srv: 98%, /var/lib/docker: 28%): OFFLINE due to disk space
[13:32:17] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11583226 (10TheDJ) There's also an issue with audio transcodes, that will require backport.  https://gerrit.wikimedia.org/r/c/medi...
[13:35:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 778147 integration-agent-docker-1043 (/: 36%, /srv: 54%, /var/lib/docker: 28%): RECOVERY disk space OK
[13:36:13] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11583251 (10TheDJ)
[13:41:33] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11583271 (10hashar) @TheDJ it is great to know someone is caring about TimedMediaHandler! 🎉 Thank you :)
[14:39:34] <wikibugs>	 06Release-Engineering-Team (Radar), 10Ceph, 06ServiceOps new, 10SRE-swift-storage, and 3 others: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11583538 (10elukey) >>! In T412951#11576712, @Scott_French wrote: > Thank you,...
[14:51:57] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11583663 (10hashar)
[14:52:37] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11583674 (10hashar) T416474 has been fixed and verified. I have left it open because some transcodings would need to be retriggere...
[15:00:12] <wikibugs>	 06Project-Admins, 10MediaWiki-Core-Tests, 06MediaWiki-Engineering: The api-testing library has no specific phab tag and its docs link to an archived one - https://phabricator.wikimedia.org/T415836#11583706 (10Krinkle) a:03Krinkle Per Mateus, we can tag create a little tag for this and link up to the MWI in...
[15:12:17] <wikibugs>	 10Gerrit, 10GitLab (Misc), 06Release-Engineering-Team (Priority Backlog 📥), 10Technical-Tool-Request, 10Wikimedia-GitHub: A tool for displaying the canonical location of all Wikimedia repositories and any known mirrors - https://phabricator.wikimedia.org/T330347#11583804 (10Aklapper) a:05brennen→03Non...
[15:18:28] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Portals: Migrate Wikimedia Portals from Gerrit to GitLab - https://phabricator.wikimedia.org/T344743#11583830 (10Aklapper) a:05Jdrewniak→03None @Jdrewniak: Removing task assignee as this open task has been assigned for more than two years - See the email...
[15:50:01] <wikibugs>	 (03CR) 10Mooeypoo: [C:03+1] "This is needed for I2a2be8f50cf5710bc , so I'd love for this to be merged/deployed soon." [integration/config] - 10https://gerrit.wikimedia.org/r/1235041 (owner: 10Jforrester)
[16:00:20] <wikibugs>	 (03PS2) 10Hashar: Zuul: [WikimediaCustomizations] Add PageViewInfo phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1235041 (owner: 10Jforrester)
[16:01:17] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Rebased to solve conflict with Ibe50f33ccc679c6f4bfaa1e9002b369b494e98d5 which introduced that extension and five dependencies ;-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1235041 (owner: 10Jforrester)
[16:03:30] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [WikimediaCustomizations] Add PageViewInfo phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1235041 (owner: 10Jforrester)
[16:06:44] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236342 (owner: 10Fomafix)
[16:08:10] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236342 (owner: 10Fomafix)
[16:10:28] <wikibugs>	 (03open) 10jelto: add repos/sre/hoarde to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/142 (https://phabricator.wikimedia.org/T414112)
[16:11:26] <wikibugs>	 (03merge) 10jelto: add repos/sre/hoarde to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/142 (https://phabricator.wikimedia.org/T414112)
[16:14:00] <wikibugs>	 (03PS1) 10Hashar: Zuul: fix typo in dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236772
[16:14:39] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend phan dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1236342 (owner: 10Fomafix)
[16:41:48] <wikibugs>	 (03open) 10volker-e: Bump Codex to 2.3.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/117 (https://phabricator.wikimedia.org/T415727)
[17:32:10] <wmf-insecte>	 Project mediawiki-core-doxygen build #17513: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17513/
[18:18:10] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11584558 (10brennen)
[18:31:39] <brennen>	 hashar: sending the usual train blocker e-mail and complaining on slack.  i'll roll the train if possible during the US window.
[18:32:56] <wmf-insecte>	 Yippee, build fixed!
[18:32:56] <wmf-insecte>	 Project mediawiki-core-doxygen build #17514: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17514/
[18:37:51] <hashar>	 brennen: of course I did not think about sending a general email!!! 😱
[18:38:17] <hashar>	 thank you!
[18:41:45] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabriactor history stream - https://phabricator.wikimedia.org/T416510#11584691 (10A_smart_kitten) [@matmarex fyi]
[18:42:24] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabriactor history stream - https://phabricator.wikimedia.org/T416510#11584698 (10Esanders) I note in the html, these are differentiated with `phui-timeline-major-event` and `phui-timeline-minor-event` classes, however the styling is nearly identical.
[18:45:51] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabriactor history stream - https://phabricator.wikimedia.org/T416510#11584713 (10Esanders) As a starting point: `css .phui-timeline-major-event a.phui-timeline-image[href="/p/gerritbot/"] ~ .phui-timeline-group .phui-timeline-core-content .transaction-comment {...
[18:46:03] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabriactor history stream - https://phabricator.wikimedia.org/T416510#11584716 (10Esanders)
[18:49:30] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabricator history stream - https://phabricator.wikimedia.org/T416510#11584734 (10SomeRandomDeveloper)
[21:25:19] <wikibugs>	 10Phabricator: Gerrit patches are easy to miss in Phabricator history stream - https://phabricator.wikimedia.org/T416510#11585153 (10matmarex) Yeah, this is my doing from {T393289}.  It would be near to have some styling that is more prominent than other timeline entries, but less prominent than human comments....