[00:01:54] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:54] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:07:09] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:12:09] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:25:53] (03PS1) 10Jdlrobson: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) [00:26:34] (03CR) 10CI reject: [V:04-1] Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [07:07:16] (03CR) 10Hashar: [C:03+2] layout.yaml: Add trusted user blankeclair [integration/config] - 10https://gerrit.wikimedia.org/r/1074966 (owner: 10Samtar) [07:09:07] (03Merged) 10jenkins-bot: layout.yaml: Add trusted user blankeclair [integration/config] - 10https://gerrit.wikimedia.org/r/1074966 (owner: 10Samtar) [07:20:35] hashar: good morning, shall we roll out https://gerrit.wikimedia.org/r/c/integration/config/+/1074480 ? [07:22:14] kostajh: \o/ :) [07:22:34] I wanted to deploy it yesterday but missed the reminder [07:22:53] (03CR) 10Hashar: [C:03+2] zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074480 (https://phabricator.wikimedia.org/T50217) (owner: 10Hashar) [07:24:10] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1074966 (owner: 10Samtar) [07:24:22] (03Merged) 10jenkins-bot: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074480 (https://phabricator.wikimedia.org/T50217) (owner: 10Hashar) [07:24:42] alright :) [07:25:16] hashar: I've proposed some wording for message to wikitech-l, on `#phpunit-cleanup-2024-edition` in Slack [07:25:41] I got banned from Slack by Slack.com due to swearing [07:25:57] (I am kidding :D ) [07:26:34] deployed [07:27:00] lol [07:27:28] kostajh: +1 on message :) [07:27:38] it is an excellent idea to announce it widely [07:27:50] remembers me I need to move forward the access request for codders [07:29:02] yeah - please :) [07:34:11] I have sent a reminder to ops list [07:59:36] (03PS4) 10Arthur taylor: Revert^2 "Add Wikibase client-only jobs to EntitySchema checks" [integration/config] - 10https://gerrit.wikimedia.org/r/1062682 (https://phabricator.wikimedia.org/T367156) [08:02:08] @hashar: can we also have another try at https://gerrit.wikimedia.org/r/1062682 today / soon? That's been languishing for a while now [08:04:58] codders: sure [08:05:02] what failed on that one? ;) [08:05:14] oh that is mentioned [08:05:38] (03CR) 10Hashar: [C:03+2] Revert^2 "Add Wikibase client-only jobs to EntitySchema checks" [integration/config] - 10https://gerrit.wikimedia.org/r/1062682 (https://phabricator.wikimedia.org/T367156) (owner: 10Arthur taylor) [08:05:49] thank you! [08:05:56] bah I did not run jjb update [08:07:14] (03Merged) 10jenkins-bot: Revert^2 "Add Wikibase client-only jobs to EntitySchema checks" [integration/config] - 10https://gerrit.wikimedia.org/r/1062682 (https://phabricator.wikimedia.org/T367156) (owner: 10Arthur taylor) [08:08:39] (03CR) 10Hashar: [C:03+2] "I have updated the Jenkins jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1062682 (https://phabricator.wikimedia.org/T367156) (owner: 10Arthur taylor) [08:08:42] codders: done! [08:08:57] you'd have to verify it is working fine [08:09:07] yup. will report back if that's working or not [08:16:44] 06Project-Admins: WTS 2024 Mini Hackathon #HackWTS2024 - https://phabricator.wikimedia.org/T375466 (10Kasyap) 03NEW [08:56:20] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10170311 (10Jelto) I reviewed the [throttling in the past 7 days](https://grafana.wikimedia... [09:28:08] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07Developer Productivity, 10MediaWiki-Platform-Team (Radar): Create a daily job running core + extension PHPUnit tests serially - https://phabricator.wikimedia.org/T372618#10170375 (10hashar) We already have [[ https://integration.wikimedia.org/ci/job/mwc... [09:28:28] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477 (10jnuche) 03NEW [09:32:54] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170425 (10jnuche) Note that the main stage that deploys to K8s normally takes around 6 minutes. Assuming a factor 10 in the time increase, that would mean an hour just to deploy that... [09:39:24] jnuche: I filed a similar k8s is slow task last week [09:39:26] https://phabricator.wikimedia.org/T374907 [09:39:46] I filed it on Tuesday morning after running what was probably the first sync of the day [09:40:35] I guess that is the same underlying issue: transferring the large images take a long time [09:42:22] looks like that yeah [09:42:33] sync-canaries-k8s is now the stage taking the brunt of downloading the images [09:43:26] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170459 (10akosiaris) > This increase in deployment times coincides with the deployment to production of the following scap change: https://gitlab.wikimedia.org/repos/releng/scap/-/me... [09:43:34] I guess the task you filed is a dupe of the one I did last week [09:43:43] which itself is a dupe of whatever is being acted on that front :D [09:44:18] which is probably that T366778 you have mentioned [09:44:18] T366778: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778 [09:44:26] anyway, Alexandros knows about it [09:51:25] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170512 (10akosiaris) {F57533835} I see some evictions happening during the deployment that could explain this, trying to correlate. [09:56:48] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10170568 (10Jelto) [09:56:58] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: Pushing mediawiki-multiversion Docker image from deploy server takes 4 minutes - https://phabricator.wikimedia.org/T341441#10170569 (10hashar) If compression / CPU is a bottleneck, and assume the images layers are already compressed, I imagine... [10:01:47] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170585 (10jnuche) It seems timeouts are currently hardcoded: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/mw-ap... [10:07:51] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170591 (10akosiaris) https://logstash.wikimedia.org/goto/69fa724990f8f554ac97601360675c79 points out the the slowest image pull was at 2m27s, most were way faster. webserver ones at... [10:34:36] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170629 (10akosiaris) >>! In T375477#10170585, @jnuche wrote: > It seems timeouts are currently hardcoded: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/... [11:11:12] 10Gerrit, 10wikitech.wikimedia.org, 07LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647#10170714 (10Ladsgroup) 05Open→03Declined Renaming shell/idm/gerrit accounts is out of the scope of wikitech SULification so I'm not sure reopening this ticket makes s... [12:22:23] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10170975 (10akosiaris) Adding 1 more data point. In the previous deployment I see also ` Liveness probe failed: command "/usr/bin/test -S /run/shared/fpm-www.sock" timed out ` for... [12:33:58] 10Continuous-Integration-Infrastructure, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10171032 (10jijiki) a:03jijiki [12:34:31] 10Scap: scap leaves mediawiki main releases in the "forward" version after a canary rollback - https://phabricator.wikimedia.org/T375497 (10akosiaris) 03NEW [12:37:01] 10Scap: scap leaves mediawiki main releases in the "forward" version after a canary rollback - https://phabricator.wikimedia.org/T375497#10171091 (10akosiaris) For what is worth, if scap was the only way helmfile commands would ever be run, the above would be totally ok. However, we can not rule out that someone... [12:40:52] (03update) 10jnuche: Set up /workspace/venvs/scap [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/90 (owner: 10dancy) [12:41:41] (03approved) 10jnuche: Set up /workspace/venvs/scap [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/90 (owner: 10dancy) [13:06:41] (03approved) 10jnuche: Establish a table for storing jobrunner status [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/454 (owner: 10dancy) [13:09:18] 10Scap, 10MW-on-K8s, 06serviceops: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10171326 (10akosiaris) [13:25:24] (03close) 10lucaswerkmeister-wmde: Report final k8s progress as 100% [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/357 [13:40:26] (03PS2) 10Dima koushha: Add SpamBlacklist to Wikibase CI setup [integration/config] - 10https://gerrit.wikimedia.org/r/1075234 [13:49:53] I'm seeing a few parallel testing failures that only appear on PHP 7.4. Should we just disable the failures for that flavour (given it's on its way out)? [15:35:11] 10Gerrit: Gerrit emails do not include comments on files that were not modified in the change - https://phabricator.wikimedia.org/T375529#10172042 (10Umherirrender) →14Duplicate dup:03T355259 [15:36:00] 10Gerrit: Gerrit notification emails are missing the content of inline comments on unchanged files - https://phabricator.wikimedia.org/T355259#10172044 (10Umherirrender) [15:36:18] 10Gerrit: Gerrit emails do not include comments on files that were not modified in the change - https://phabricator.wikimedia.org/T375529#10172071 (10Lucas_Werkmeister_WMDE) Thanks, not sure why I couldn’t find the existing task. [15:38:50] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994#10172077 (10Krinkle) [15:38:57] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#10172078 (10Krinkle) [15:46:52] 10Continuous-Integration-Config: CI trying to install master Vector for FR? - https://phabricator.wikimedia.org/T375533 (10Reedy) 03NEW [15:50:29] 10Continuous-Integration-Config, 07Code-Health: Code Health trying to install master Vector for FR? - https://phabricator.wikimedia.org/T375533#10172137 (10Reedy) [15:53:38] (03PS1) 10Jforrester: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) [15:54:40] (03CR) 10CI reject: [V:04-1] Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [15:56:08] 10Release-Engineering-Team (Priority Backlog 📥), 10Structured Data Engineering, 06Structured-Data-Backlog, 10Wiki-Setup (Delete / Redirect): Close TestCommons from production  - https://phabricator.wikimedia.org/T213295#10172151 (10Jdforrester-WMF) 05Declined→03Resolved [15:56:39] (03PS2) 10Jdlrobson: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) [15:59:59] (03PS2) 10Jforrester: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) [16:01:08] (03CR) 10CI reject: [V:04-1] Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [16:06:22] (03PS3) 10Jforrester: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) [16:07:29] (03CR) 10CI reject: [V:04-1] Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [16:08:33] I give up, Python linting is a disaster. [16:10:58] it literally doesn't like () in comments? [16:11:01] That's... helpful... [16:11:06] I have no idea. [16:11:13] or... end of line? [16:11:17] Maybe? [16:11:22] it's pointing at the trailing ) [16:11:36] Yes, but before that it was pointing at the ? of ?). [16:11:37] There's definitely () in use above [16:11:39] # But practically speaking, Parsoid is (for the moment) an extension. [16:11:41] Yeah. [16:11:59] Or does it just not like the comments inline between the if conditions? [16:12:11] Oh, if so that's ultra-sucky. [16:12:35] It definitely feels like a linter bug if that is the case [16:12:43] Sure, it may not be desired, but it shouldn't be resulting in a syntax error [16:12:56] Aha, I can wrap it in ()s to make it happy. [16:13:07] (03PS4) 10Jforrester: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) [16:13:32] Anyway, getting it passing lint is just the first step; I wanted input on my proposal before JFDIing and deploying. :-) [16:13:43] hi all, i will try to test https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1075211 on the beta cluster. if anything breaks horribly in the next ten minutes, it's probably my fault [16:13:55] MatmaRex: Most things are. [16:14:01] lol [16:14:03] Do you need the beta update jobs disabled? [16:14:16] Or just manually editing it on deployment-deploy03 or whatever? [16:14:28] James_F: If it's broken and we (understandably) have no intention of fixing it... It seems a reasonble thing to do [16:14:43] no, i just need a few minutes, then the jobs can revert it [16:14:44] (03CR) 10CI reject: [V:04-1] Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [16:15:15] Reedy: It was deployed a couple of hours ago, they might have views. [16:16:08] Reedy: But yeah, if we just undeploy PHP 7.4 from prod and CI soon it'll save a lot of work. ;-) [16:25:13] (03CR) 10Subramanya Sastry: "I'll cherry-pick on the server and rerun tests." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [16:40:32] FWIW, python linting looks like it's mad about `not "php74" in` for `"php74" not in` [16:41:08] Aha, that's actually sensible. [16:41:14] Though not what the original issue was. [16:41:31] (03PS5) 10Jforrester: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) [16:43:33] Gosh, it actually passes CI now. [16:44:59] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [16:45:22] joining late to the party - so parallel testing and php7.4 was no bueno? sorry was out of electricty and my irc boucer died and didn't reconnect [16:48:08] pmiazga: It seems like all the random failures are in PHP 7.4, yes. I was thinking it'd save a bunch of disruption if we skipped them for now, and maybe fixed them later (or just dropped PHP 7.4). [16:49:59] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [16:51:10] We still need to run PHP 7.4 for WMF production though [16:51:14] yeah, nice find James_F, definitely if we can make the transition less painful I would say go for it :) [16:51:19] Reedy: For now. [16:51:23] Yeah [16:51:30] "Later" is doing a lot of work in my sentence, to be fair. [16:51:34] :) [16:51:39] (03CR) 10Jforrester: [C:03+2] Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [16:51:44] Let's do it, then. [16:52:28] It's still a net win having it enabled on the $otherbranches [16:52:36] Yup. [16:52:55] $otherjobs, same branches (master and wmf/*) [16:53:27] (03Merged) 10jenkins-bot: Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075264 (https://phabricator.wikimedia.org/T50217) (owner: 10Jforrester) [16:53:34] !log Zuul: Disable parallel PHPUnit in Quibble for PHP 7.4 jobs, for T50217 [16:53:34] and fundraising [16:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:53:39] T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel - https://phabricator.wikimedia.org/T50217 [16:54:47] Project beta-code-update-eqiad build #514589: 04FAILURE in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/514589/ [16:55:00] gj MatmaRex [16:55:11] Ha. [16:55:28] xD [16:55:42] fatal: HEAD does not point to a branch [16:55:43] indeed [17:01:18] (03PS1) 10Reedy: zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 [17:02:38] (03CR) 10CI reject: [V:04-1] zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 (owner: 10Reedy) [17:03:29] (03PS2) 10Reedy: zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 [17:04:23] Reedy: Is there a Phab task? [17:04:39] kinda https://gerrit.wikimedia.org/r/1059857 [17:04:49] Oh, fun. [17:04:52] Project beta-code-update-eqiad build #514590: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/514590/ [17:10:43] (03CR) 10Dreamy Jazz: [C:03+1] zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 (owner: 10Reedy) [17:11:00] i am struggling somewhat with my testing. i restored mediawiki-staging now, so the next job should succeed. [17:11:13] (03PS3) 10Reedy: zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 (https://phabricator.wikimedia.org/T371267) [17:11:40] 10Continuous-Integration-Config: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541 (10Reedy) 03NEW [17:11:50] * apergos peeks in and watches the "fun" [17:12:20] Reedy: Yes! But I couldn't work out how to make the injection code happy. [17:12:35] 10Continuous-Integration-Config: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10172422 (10Reedy) [17:12:40] we should just rewrite it in PHP and... [17:13:05] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10172428 (10Jdforrester-WMF) [17:13:31] Reedy: It's Jenkins, so we should re-write in Groovy, right? [17:14:04] "(I wouldn't be" – cool, gnomic starts to task descriptions. ;-) [17:15:06] (03CR) 10Reedy: [C:03+2] zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 (https://phabricator.wikimedia.org/T371267) (owner: 10Reedy) [17:15:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10172432 (10Reedy) [17:15:42] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10172444 (10Reedy) [17:16:27] (03Merged) 10jenkins-bot: zuul/parameter_functions.py: Load CheckUser for CentralAuth CI and phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1075274 (https://phabricator.wikimedia.org/T371267) (owner: 10Reedy) [17:16:27] Yippee, build fixed! [17:16:27] Project beta-code-update-eqiad build #514591: 09FIXED in 3 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/514591/ [17:17:42] !log Reloading Zuul to deploy https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/514591/ [17:17:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:17:46] DAMN IT [17:17:52] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1075274 [17:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:19:18] lol [17:19:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10172451 (10Reedy) Maybe even reusing `layout.yaml` may make sense... [17:34:03] Reedy: Have the patch mostly written. [17:34:09] ooh <3 [17:34:48] * Reedy waits for the but [17:34:55] But I'm in meetings. [17:38:02] 10Continuous-Integration-Config, 07Code-Health: Code Health trying to install master Vector for fundraising mw core branch? - https://phabricator.wikimedia.org/T375533#10172495 (10Reedy) [17:46:25] (03PS1) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [17:47:24] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [17:47:39] Untested. And CI is unhappy. [18:00:44] (03CR) 10Thcipriani: [C:04-1] Zuul: Split out deps and phan_deps to YAML files (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:40:10] (03PS1) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) [18:44:14] (03PS1) 10Reedy: zuul/parameter_functions.py: Remove parsoid extension remapping [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 [18:45:29] (03CR) 10CI reject: [V:04-1] zuul/parameter_functions.py: Remove parsoid extension remapping [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 (owner: 10Reedy) [18:45:41] (03PS1) 10SD0001: Add Scribunto as a phan dependency for Gadgets extension [integration/config] - 10https://gerrit.wikimedia.org/r/1075301 (https://phabricator.wikimedia.org/T71441) [18:46:34] (03PS2) 10Reedy: zuul/parameter_functions.py: Remove parsoid extension remapping [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 [18:46:49] (03PS2) 10SD0001: Add Scribunto as a phan dependency for Disambiguator extension [integration/config] - 10https://gerrit.wikimedia.org/r/1075301 (https://phabricator.wikimedia.org/T71441) [18:48:01] (03PS2) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [18:48:01] (03CR) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:48:23] (03CR) 10CI reject: [V:04-1] zuul/parameter_functions.py: Remove parsoid extension remapping [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 (owner: 10Reedy) [18:49:05] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:49:23] (03PS3) 10Reedy: zuul/parameter_functions.py: Remove parsoid extension remapping [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 [18:49:26] (03PS3) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [18:50:44] (03PS4) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [18:50:48] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:51:55] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:52:52] interesting... [18:52:53] 10Continuous-Integration-Config, 07PHP 8.1 support: Configure CI to test the Wikimedia production gate code against PHP 8.1 as well as PHP 7.4 - https://phabricator.wikimedia.org/T320818#10172728 (10kostajh) @hashar I think this task is done? [18:53:34] (03CR) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [18:54:05] 10Continuous-Integration-Config, 07PHP 8.1 support: Configure CI to test the Wikimedia production gate code against PHP 8.1 as well as PHP 7.4 - https://phabricator.wikimedia.org/T320818#10172729 (10Jdforrester-WMF) >>! In T320818#10172728, @kostajh wrote: > @hashar I think this task is done? It was done, has... [19:00:44] (03CR) 10Thcipriani: Zuul: Split out deps and phan_deps to YAML files (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:01:13] (03CR) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:01:23] (03CR) 10CI reject: [V:04-1] DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) (owner: 10Kosta Harlan) [19:01:45] (03PS2) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375551) [19:02:21] (03PS1) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075303 (https://phabricator.wikimedia.org/T375551) [19:02:58] (03PS3) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) [19:19:53] (03CR) 10CI reject: [V:04-1] DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075303 (https://phabricator.wikimedia.org/T375551) (owner: 10Kosta Harlan) [19:20:34] (03CR) 10CI reject: [V:04-1] DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) (owner: 10Kosta Harlan) [19:24:57] (03CR) 10Thcipriani: Zuul: Split out deps and phan_deps to YAML files (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:38:47] (03PS5) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [19:39:49] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:43:26] (03PS6) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [19:44:57] (03CR) 10Reedy: Zuul: Split out deps and phan_deps to YAML files (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:45:24] (03PS7) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [19:45:27] Everyone's a critic. :-) [19:45:48] en-gb or en-us? yes. [19:46:20] Shall we try it out and see what breaks? [19:47:02] the non voting change is failing with the new exception [19:47:05] very helpfully [19:47:21] Oh, hmm. [19:48:02] (03PS8) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [19:48:20] It's not flailing around, it's *debugging*. [19:49:11] >20:49:02 Exception: ZUUL_DIR could not be set, environment not recognised; I'm running in: /src/integration/config [19:49:12] heh [19:49:18] Bleh. [19:49:25] TBF they should all really be doing that. [19:49:28] (03CR) 10CI reject: [V:04-1] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:49:43] James_F: Reedy: thanks for the parallel rollback :) [19:49:49] Of course. [19:50:24] I felt I had enough push back and was too conservative [19:50:31] (03PS9) 10Jforrester: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) [19:50:39] hashar: Observe ^^^ [19:50:40] when it's relatively easy to revert... [19:50:45] so I went to approve it given kosta/arthur di plenty of work on it already [19:50:51] phan_deps should disappear [19:50:52] Sometimes it's easier to just see what happens in reality [19:51:07] Reedy: yeah that is very true [19:51:34] (03PS3) 10Jdlrobson: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) [19:51:45] for phan_depsn I think we have a task somewhere which is to have Phan to clone the requirements expressed in `extension.json` [19:51:52] anyway too many tasks [19:51:57] Yeah, but we don't have that yet... [19:52:12] The lists are incomplete... There's no ways to do soft depends etc [19:52:15] that also implies supporting optional requirements in the extension registry [19:52:57] and writing some code to be able to clone/fetch/checkout the CI prepared patches [19:53:01] aka port Zuul cloner to PHP [19:53:04] pff [19:53:06] Which is not trivial. [19:53:09] oh [19:53:11] well [19:53:20] Shall we deploy it and see what breaks? [19:53:24] does it fit in an OKR / PROMO package? [19:53:54] the serious thing is it is low effort to add the dependencies in the yaml file [19:54:02] Yes. [19:54:20] (03PS4) 10Jdlrobson: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) [19:54:36] definitely less effort than making changes to extension registry or writing system code in PHP :D [19:54:48] hashar: OK for me to deploy? [19:54:54] I'd rather port MediaWiki to Python and see Reedy fainting at the sight of flake8 errors [19:54:56] Easily tested and reverted. [19:55:10] If we port to Python, I'm looking for a new gig. :-) [19:55:18] man [19:55:34] it could be worse! [19:55:43] it could be JavaScript! [19:56:01] for the split of dependencies, yes go for it [19:56:09] if that makes it easier to differentiate them :] [19:56:27] (03CR) 10Jforrester: [C:03+2] Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:56:29] beware of import/require. I am not sure how it would though [19:56:36] Yeah. [19:57:45] (03Merged) 10jenkins-bot: Zuul: Split out deps and phan_deps to YAML files [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [19:58:01] * hashar screams at the amount of IO system calls involved :D [19:58:13] !log Zuul: Split out deps and phan_deps to YAML files, for T375541 [19:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:58:16] T375541: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541 [19:59:37] * James_F hunts for what's now broken. [19:59:45] # HACK: This is horrible, but __file__ isn't defined. [19:59:47] nice [20:00:02] I bow to thcipriani. [20:00:06] it is not imported [20:00:14] it is executed within some context which would lack a file [20:00:32] cause it is something like `eval( "some string of code" )` [20:00:34] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium/42050/console shows the right clone list. [20:00:44] hashar: :oh-no: [20:02:52] with open(fn) as _f: [20:02:52] code = compile(_f.read(), fn, 'exec') [20:02:53] six.exec_(code, config_env) [20:03:12] that is the code to load the zuul/parameters_functions.py [20:03:20] which is well [20:03:22] hmm [20:03:29] ignore it :) [20:03:48] (03CR) 10Thcipriani: "One thought here after reading zuul code." [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [20:04:27] ^ random thought that might be fancier [20:05:50] print(error) [20:05:52] :D [20:06:20] I wonder where the error message ends up being shown. Maybe that is in the daemon stdout and thus end up in journalctl [20:06:31] anyway thanks for hacking stuff on that front! [20:07:19] (03CR) 10Subramanya Sastry: Workaround various other known issues and force loading of images (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [20:12:32] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Wikisource] Fix typo in phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1075312 [20:12:44] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Wikisource] Fix typo in phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1075312 (owner: 10Jforrester) [20:13:10] heh [20:13:59] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikisource] Fix typo in phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1075312 (owner: 10Jforrester) [20:14:06] !log Zuul: [mediawiki/extensions/Wikisource] Fix typo in phan deps [20:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:14:40] * hashar is off [20:16:55] (03PS5) 10Jdlrobson: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) [20:17:00] (03CR) 10Jdlrobson: Workaround various other known issues and force loading of images (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [20:17:58] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Disambiguator] Add Scribunto as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075301 (https://phabricator.wikimedia.org/T71441) (owner: 10SD0001) [20:20:38] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Wikibase] Add SpamBlacklist as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075234 (owner: 10Dima koushha) [20:39:50] (03PS2) 10Jforrester: [WIP] Docker: Re-build PHP 7.4 & 8.1+ with php-uuid [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) [22:18:11] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Move `dependencies` and `phan_dependencies` to separate YAML/JSON file from zuul/parameter_functions.py - https://phabricator.wikimedia.org/T375541#10173479 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [22:19:00] (03CR) 10Jforrester: [C:03+2] "That'd leave us with more Python files, which we're trying to avoid, but yes." [integration/config] - 10https://gerrit.wikimedia.org/r/1075279 (https://phabricator.wikimedia.org/T375541) (owner: 10Jforrester) [22:58:21] 10Gerrit, 10wikitech.wikimedia.org, 07LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647#10173636 (10Kizule) >>! In T260647#10170714, @Ladsgroup wrote: > Renaming shell/idm/gerrit accounts is out of the scope of wikitech SULification so I'm not sure reopening...