[02:17:10] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Tgr) >>! In T338384#8911697, @taavi wrote: > Is there a suitable hook in CentralAuth alrea... [05:42:08] 10Release-Engineering-Team (Deployment-Blockers), 10Reference Previews, 10WMDE-TechWish, 10MW-1.41-notes (1.41.0-wmf.30; 2023-10-10), 10WMDE-TechWish-Maintenance-2023: ReferencePreviews: Final round of manual tests before full rollout - https://phabricator.wikimedia.org/T345833 (10WMDE-Fisch) [05:49:29] 10Phabricator: The content of the "Activity" column on Phabricator's homepage is incorrectly set to display none. - https://phabricator.wikimedia.org/T346169 (10valerio.bozzolan) (I cannot see the image F37711476 btw, please edit and wide the Visibility) [06:03:41] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10ArielGlenn) Um there is no Thurs Oct 8. There is Thurs Oct 5 (today) and Thurs Oct 12, 19, 26... wonder if you meant any of these? [06:22:15] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should log linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10kostajh) [06:48:29] 10Phabricator (Upstream), 10Upstream: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10valerio.bozzolan) [06:48:48] 10Phabricator (Upstream), 10Upstream: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10valerio.bozzolan) Thanks! Very interesting legacy heritage. Sometime the Tab Panels had 2+ tabs opened. Now you have zero tabs opened. Tha... [06:52:32] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should log linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Aklapper) >>! In T338384#9226941, @Tgr wrote: > wouldn't you want to block the user on wikite... [08:01:38] GitLab maintenance window starts in one hour: 09:00-13:00 UTC. We will switch GitLab from eqiad to codfw. See T345531 [08:01:38] T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 [08:16:38] 10WikimediaDebug, 10MW-on-K8s, 10observability: Excimer UI profile lost when requested from mw-on-k8s - https://phabricator.wikimedia.org/T347926 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi >>! In T347926#9226441, @Krinkle wrote: > @fgiunchedi For baremetal, it is intentional that this is not limit... [08:27:52] (03CR) 10Hashar: [C: 03+2] Require tox v4 and remove skipsdist/use_develop [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) (owner: 10Hashar) [08:43:29] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation cleanup/completion/improvement - https://phabricator.wikimedia.org/T342317 (10KBach) 05Open→03In progress a:03KBach I'll work on this - currently in the research/discovery phase. [08:44:00] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 (10hashar) I have manually edited https://integration.wikimedia.org/ci/job/pywikibot-core-tox-docker/ to change the... [08:44:25] 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/46 [gitlab/switchover] Up... [08:45:37] (03Merged) 10jenkins-bot: Require tox v4 and remove skipsdist/use_develop [integration/quibble] - 10https://gerrit.wikimedia.org/r/960082 (https://phabricator.wikimedia.org/T346238) (owner: 10Hashar) [08:55:55] 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10Jelto) [08:59:21] GitLab maintenance starting now - T345531 [08:59:21] T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 [09:01:22] 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikime... [09:04:27] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation cleanup/completion/improvement - https://phabricator.wikimedia.org/T342317 (10KBach) p:05Triage→03Medium [10:01:20] hashar: could you take a look at https://phabricator.wikimedia.org/T348176. i think you looked at this, and fixed it before [10:01:44] i have chcked pcc-worker1002 and that has free space i think this may be on the jenkins side? [10:02:34] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Puppet CI, 10SRE: PCC failing with "No space left on device" - https://phabricator.wikimedia.org/T348176 (10jbond) @hashar could you check if this is on the jenkins side pcc-worker1002 looks healthy to me [10:32:20] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10dr0ptp4kt) [10:33:54] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10dr0ptp4kt) >>! In T347089#9227026, @ArielGlenn wrote: > Um there is no Thurs Oct 8. There is Thurs Oct 5 (today) and Thurs Oct 12, 19, 26... wonder if you meant any... [10:49:54] jbond: checking :) [10:50:12] hashar: no worries i found the issue [10:50:21] was inode susagen in the pcc folder [10:50:21] out of inodes ... doh [10:50:32] *usage [10:50:43] last time I think some garbage collecting cron job was missing to discard old compilations [10:51:01] then that is on `/` , isn't pcc writting to `/srv`? [10:51:14] yes this directory is normally cleaned up by pcc but i thinik if a job ios canceled in jenkins then the files persist [10:51:26] ahh [10:51:26] yes but srv is nt a seperate mount [10:53:15] if a job is cancelled, Jenkins should send a SIGTERM to the process (or to each of the process in the process group , I can't remember how it does it) [10:53:22] then I think it eventually SIGKILL them [10:53:45] maybe there can be a `trap` added [10:53:46] ahh ok ill create a task to add a signal handeler to pcc [10:54:23] or the Jenkins job can potentially remove the artifacts once the build has completed/canceled, but I am not sure that behaves properly when a job is canceled :/ [10:55:32] g3.cores4.ram8.disk20 , so yeah everything on asingle 20G partition [10:56:46] potentially some storage volumes can be created and attached to the workers, looks like that is how pcc-worker1001 is setup [10:58:54] ah no the volume is not mounted [10:59:09] hashar: most out put kgoes to a shared nfs partition [10:59:24] this is more like a scratch dir i can clean it either with pcc or a tiomer or both [10:59:46] we can be quite bruital an just delete everything older then a day in that dir [11:01:14] +1 :) [11:01:36] and potentially trigger a cleanup when receiving SIGTERM [11:01:42] exactly [11:03:01] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 18 - https://phabricator.wikimedia.org/T331181 (10hashar) 05Open→03Resolved node18 jobs have been created by https://gerrit.wikimedia.org/r/c/integration/config/+/954333 [11:03:03] 10Fresh, 10Growth-Team, 10GrowthExperiments: Add Fresh support for Node.js 18 (with npm 9) - https://phabricator.wikimedia.org/T337647 (10hashar) [11:03:05] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10hashar) [11:03:32] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 20 - https://phabricator.wikimedia.org/T343826 (10hashar) 05Open→03Resolved node20 jobs have been created by https://gerrit.wikimedia.org/r/c/integration/config/+/954333 [11:03:34] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827 (10hashar) [11:46:35] 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikime... [11:49:12] Gitlab is back, maintenance done [12:27:03] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10LSobanski) [12:44:52] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10Muhammad_Yasser_Jazirahly_WMDE) [12:46:46] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10Lucas_Werkmeister_WMDE) There are also a lot of warnings earlier (copied from [another build](https://integration.wikimedia.org/ci/job/mwgate-node16-docker/74729/con... [12:47:00] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5), 10ci-test-error (WMF-deployed Build Failure): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10Lucas_Werkmeister_WMDE) [13:33:02] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10Jelto) [13:45:43] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5), 10ci-test-error (WMF-deployed Build Failure): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10hashar) That is the npm cache used by CI which is corrupted somehow, possibly due to a race condition. I have never f... [13:48:37] (03PS1) 10Hashar: dockerfiles: keep pip and wheels in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/963735 [13:51:12] (03CR) 10Hashar: [C: 03+2] dockerfiles: keep pip and wheels in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/963735 (owner: 10Hashar) [13:52:27] (03Merged) 10jenkins-bot: dockerfiles: keep pip and wheels in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/963735 (owner: 10Hashar) [13:59:45] (03PS1) 10Hashar: jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963738 [14:12:14] (03CR) 10Hashar: [C: 03+2] jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963738 (owner: 10Hashar) [14:13:46] (03Merged) 10jenkins-bot: jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963738 (owner: 10Hashar) [14:18:24] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T347080 (10Lucas_Werkmeister_WMDE) [14:20:19] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5), 10ci-test-error (WMF-deployed Build Failure): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10Lucas_Werkmeister_WMDE) Seems to be working again, thanks! [14:37:13] 10Release-Engineering-Team, 10Wikibase Product Platform Team WPP (Sprint 5), 10ci-test-error (WMF-deployed Build Failure): Wikibase CI is broken - https://phabricator.wikimedia.org/T348243 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03hashar [16:37:20] !log brion batch-running TimedMediaHandler requeueTranscodes.php to clean up old VP8 WebM files [16:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:40:57] note that this cleanup may remove many gigs of video transcodes from swift :D [16:44:13] bvibber: I can't wait for the "deletion rate from swift is too high" from the monitoring [16:44:32] Also, did you mean to do this in #wikimedia-operations ? ;) [16:46:59] whoops [20:31:05] It's looking like it isn't supported but I wanna make sure: Blubberfiles do not support venvs with the python builder, correct? [20:35:18] Not yet but that has been requested. [20:35:54] thanks!