[00:00:17] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9667606 (10tstarling) Legoktm uploaded all MediaWiki tarballs from releases.wikimedia.org to the Internet Archive [[https://archive.org/details/mediawiki-archive-2018|in 2018]]. I should be able to recov... [00:28:35] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9667655 (10Pppery) [00:28:36] 10MediaWiki-Releasing, 10MediaWiki-General: 141.19 and 1.20 download directories only provides latest dot releases - 14https://phabricator.wikimedia.org/T60172#9667653 (10Pppery) →14Duplicate dup:03T190369 [00:37:48] 10Phabricator maintenance bot: #patch-for-review tags are not being removed from tasks - https://phabricator.wikimedia.org/T361074#9667665 (10Ladsgroup) There are many cases that the bot errs on the side of not removing the tag, e.g. there has been many cases that people added the tag because of github PR and as... [01:38:33] 10Phabricator maintenance bot: 14#patch-for-review tags are not being removed from tasks - 14https://phabricator.wikimedia.org/T361074#9667821 (10matmarex) 05Open→03Invalid 14Hmm, fair enough. [01:39:58] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9667824 (10tstarling) OK, well if they were already missing in 2014, I'm not going to find it in a 2018 archive. [03:12:04] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9667875 (10tstarling) It's unlikely the Internet Archive or any crawler would have files that were generated in September 2013 and reported missing in December 2013. Maybe community members would have th... [04:39:38] (03PS1) 10MusikAnimal: doc: add CodeMirror to opensource.yaml [integration/docroot] - 10https://gerrit.wikimedia.org/r/1015209 (https://phabricator.wikimedia.org/T359986) [05:33:49] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9667967 (10hashar) @csteipp has left the foundation and his account here is disabled. I guess it is therefore not going to receive any notification. releases.wikimedia.org got created in 2014 for *new... [06:09:29] 10MediaWiki-Releasing: Big holes in the MediaWiki release archive - https://phabricator.wikimedia.org/T190369#9668008 (10hashar) I have pushed some 1.19.8 material to https://people.wikimedia.org/~hashar/T190369/ I have found them from a `mediawiki-1.19.8-1.scs.el7.centos.src.rpm` package. I haven't verified t... [09:29:36] GitLab needs a short maintenance break in one hour [10:05:13] 10Release-Engineering-Team (Radar), 06Tech-Docs-Team, 07Documentation: 14Create repository with configuration files for common CI use cases - 14https://phabricator.wikimedia.org/T352273#9668174 (10KBach) 05In progress→03Resolved [10:05:23] 10Release-Engineering-Team (Radar), 06Tech-Docs-Team, 07Documentation: 14Write tutorials for common CI tasks - 14https://phabricator.wikimedia.org/T352274#9668192 (10KBach) 05Open→03Declined 14Deemed out of scope and redundant. Declining. [10:07:37] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Filburt - https://phabricator.wikimedia.org/T361191 (10Filburt) 03NEW [10:14:34] 10Release-Engineering-Team (Radar), 06Tech-Docs-Team, 07Documentation: 14Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - 14https://phabricator.wikimedia.org/T342317#9668414 (10KBach) 05In progress→03Resolved [10:33:30] GitLab upgrade done [10:34:06] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: upload.wikimedia.beta.wmflabs.org: cannot find server - https://phabricator.wikimedia.org/T359768#9668575 (10AlexisJazz) >>! In T359768#9656733, @jcrespo wrote: > I'm afraid SREs cannot help here- while it is ATS returning the error to the end user, i... [10:45:58] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: upload.wikimedia.beta.wmflabs.org: cannot find server - https://phabricator.wikimedia.org/T359768#9668656 (10Aklapper) >>! In T359768#9668575, @AlexisJazz wrote: > So nobody is responsible for this. See {T215217} and its subtasks for the general prob... [11:58:23] (03PS1) 10Cparle: Remove MachineVision - it is sunsetted [integration/config] - 10https://gerrit.wikimedia.org/r/1015298 (https://phabricator.wikimedia.org/T352884) [11:58:58] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9668916 (10Clement_Goubert) [11:59:27] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9668917 (10Clement_Goubert) [12:07:26] (03CR) 10Cparle: "Not sure if this should happen before or after dropping the extension from releng/release" [integration/config] - 10https://gerrit.wikimedia.org/r/1015298 (https://phabricator.wikimedia.org/T352884) (owner: 10Cparle) [12:13:35] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9668960 (10Clement_Goubert) [12:30:31] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.42.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T360156#9668974 (10Func) [12:45:04] 12:43:06 - https://github.com/composer/installers.git [12:45:04] 12:43:06 Cloning into bare repository '/cache/composer/vcs/https---github.com-comp [12:45:04] 12:43:06 oser-installers.git'... [12:45:05] 12:43:06 fatal: unable to access 'https://github.com/composer/installers.git/': Fa [12:45:05] 12:43:06 iled to connect to github.com port 443: Connection timed out [12:45:06] lol [12:45:54] Numerous jobs failing with that :( [12:49:10] https://www.githubstatus.com/ "All Systems Operational" [12:53:58] (03PS2) 10Hashar: Remove MachineVision - it is sunsetted [integration/config] - 10https://gerrit.wikimedia.org/r/1015298 (https://phabricator.wikimedia.org/T352884) (owner: 10Cparle) [12:54:54] (03CR) 10Hashar: [C:04-1] "I have added an entry to mark it as archived." [integration/config] - 10https://gerrit.wikimedia.org/r/1015298 (https://phabricator.wikimedia.org/T352884) (owner: 10Cparle) [12:56:58] Reedy: taavi: Krinkle mentioned it last night (roughly 3am UTC in _security) [12:58:07] I imagine that is going to be a long tail of debugging [12:58:15] but I am busy now bisecting python 2.7 :/ [13:04:42] (03PS1) 10Reedy: layout.yaml: Add Philipnelson99 to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1015322 [13:05:11] (03CR) 10Reedy: [C:03+2] layout.yaml: Add Philipnelson99 to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1015322 (owner: 10Reedy) [13:06:36] (03Merged) 10jenkins-bot: layout.yaml: Add Philipnelson99 to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1015322 (owner: 10Reedy) [13:07:01] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1015322 [13:07:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:32:24] so one problem solved [13:32:33] paramiko does not recognize old ssh keys :D [13:36:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 07Zuul, 06collaboration-services: Test zuul under Bullseye with contint1003 - https://phabricator.wikimedia.org/T361224 (10hashar) 03NEW [13:56:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 07Zuul, 06collaboration-services, 13Patch-For-Review: Test zuul under Bullseye with contint1003 - https://phabricator.wikimedia.org/T361224#9669320 (10hashar) Looks like some of the basics are working ;) [14:10:11] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T360157#9669377 (10Jdforrester-WMF) ##### Risky Patch! 🚂🔥 * **Change**: https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/67 * **Summary**:... [14:50:43] is it expected that the gate only runs jobs per repo? https://integration.wikimedia.org/zuul/. Seems that only one check user is running and the rest is backed up. [14:52:23] paladox: I count 4 changes in progress [14:52:43] the gate-and-submit limits the number of changes that have jobs running [14:52:48] oh [14:52:56] there is a window [14:53:03] starts at 2 change I think [14:53:10] then each time a change pass the limit is raised [14:53:29] and whenever a change fails (and the queue is rearranged), the window is halved (iirc) [14:54:51] https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.manager has an overview [14:55:00] we only have dependent and independent ones [14:55:21] the window system is described at https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.window [15:51:59] Ref https://phabricator.wikimedia.org/T361177 T361177 [15:52:00] T361177: Incorrect namespace displayed as title on a zh.wikivoyage page - https://phabricator.wikimedia.org/T361177 [15:52:30] This is a train blocker that warrants immediate group rollback to wmf23 for all wikis including group0 [15:52:45] I'm working on fixing the patch now [15:53:13] We are conflating all wikis in the same apcu cache prefix. Impact too broad to reason about. [15:54:27] jeena: thcipriani ^ [15:57:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06serviceops: Decommission integration.mediawiki.org - https://phabricator.wikimedia.org/T361250 (10hashar) 03NEW [15:59:06] Krinkle jeena thcipriani: i can roll back [16:01:36] eek. Thanks brennen [16:02:12] We had a patch already. But the owner went OoO between first and second patch.. [16:02:30] rollback's in progress. [16:02:35] Thanks! [16:03:43] Thanks brennen [16:03:50] sure thing [16:28:57] thcipriani: maybe that is related to the invalid title issue we witnessed on zh.wikitionary during the train triage [16:29:07] well who knows really [16:33:17] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 06collaboration-services: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9670168 (10CodeReviewBot) thcipriani updated https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/69 a... [16:33:18] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 06collaboration-services: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9670169 (10CodeReviewBot) thcipriani merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/69 ad... [16:38:18] 10GitLab (Account Approval), 06Release-Engineering-Team: 14Requesting GitLab account activation for Filburt - 14https://phabricator.wikimedia.org/T361191#9670191 (10brennen) 05Open→03Resolved [16:52:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep (beta cluster): Code stewardship request - https://phabricator.wikimedia.org/T215217#9670283 (10AlexisJazz) Beta cluster is actively used to test the [[https://github.com/commons-app/apps-android-co... [17:05:11] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: upload.wikimedia.beta.wmflabs.org: cannot find server - https://phabricator.wikimedia.org/T359768#9670378 (10AlexisJazz) ;-( https://commons.wikimedia.beta.wmflabs.org/w/thumb.php?f=Example.jpg&w=220 doesn't work either, but returns instantly instead... [17:27:03] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: upload.wikimedia.beta.wmflabs.org: cannot find server - https://phabricator.wikimedia.org/T359768#9670454 (10thcipriani) This may be related to T360595 I know the caching hosts use varnishkafka and if kafka does not have a valid ssl cert then...maybe... [17:40:52] (03PS1) 10Krinkle: jjb: document mediawiki-core-doxygen-docker cronjob and task [integration/config] - 10https://gerrit.wikimedia.org/r/1015360 [17:41:46] (03PS1) 10Krinkle: zuul: Enable CI for mediawiki/gadgets/RTRC [integration/config] - 10https://gerrit.wikimedia.org/r/1015361 [17:42:52] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations, 10observability, and 2 others: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations' - https://phabricator.wikimedia.org/T360595#9670484 (10thcipriani) >>! In T360595#9665327, @colewhite wrote: >>>!... [18:32:10] 10GitLab, 06collaboration-services: gitlab: enforce 2fa for admins - https://phabricator.wikimedia.org/T361277 (10Dzahn) 03NEW [18:32:51] 10GitLab, 06collaboration-services: gitlab: enforce 2fa for admins - https://phabricator.wikimedia.org/T361277#9670756 (10Dzahn) [18:56:32] 10Phabricator (Upstream), 07Upstream: Allow "lipsum" data generation tool to create projects with empty descriptions - https://phabricator.wikimedia.org/T355966#9670891 (10Aklapper) 05In progress→03Stalled [18:57:33] 10Phabricator (Upstream), 07Upstream: Exception: Map returned by "newPagingMapFromCursorObject()" in class "ManiphestTaskQuery" omits required key - https://phabricator.wikimedia.org/T344241#9670897 (10Aklapper) 05Open→03Stalled [18:57:46] 10Diffusion, 10Phabricator, 10Wikimedia-GitHub: 14Three repositories mirrored by Diffusion from GitLab into both "Toolforge" and "Wikimedia" on GitHub - 14https://phabricator.wikimedia.org/T349610#9670894 (10Aklapper) 05Open→03Declined a:05Aklapper→03None 14Thanks, let's keep it then [18:58:47] 10Phabricator (Upstream), 07Upstream: Allow disabling Audit application - https://phabricator.wikimedia.org/T330794#9670901 (10Aklapper) 05Open→03Stalled [18:59:08] 10Phabricator (Upstream), 07Upstream: Visually indicate when a Phabricator user is new (Welcome culture) - https://phabricator.wikimedia.org/T64324#9670903 (10Aklapper) a:05Aklapper→03None [19:12:31] I did the envoy SSL cert replacement on contint* just now. [19:12:38] https://integration.wikimedia.org/ is still up [19:12:55] removed integration.mediawiki.org though [19:13:01] per our meeting earlier today [19:13:19] but that also changes nothing because it is a rewrite rule on appservers [19:13:46] it is a clean up though and we can now delete a bunch of files in 3 repos [19:32:53] mutante: nice, thanks for the cleanup [19:33:18] :) [19:33:39] thcipriani: and thank you for amending and merging the deploy window for gitlab [19:34:53] sure thing, the script that generates the deployment calendar should run tomorrow night (for me, 2am-ish utc Saturday) [19:35:13] ok, cool! [19:35:16] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): 14Update Integration project puppetmaster - 14https://phabricator.wikimedia.org/T360461#9671030 (10Andrew) 05Open→03Resolved a:03Andrew [19:50:37] 10Phabricator (Upstream), 07Upstream: One Phabricator task displayed in two workboard columns simultaneously - https://phabricator.wikimedia.org/T335443#9671093 (10Aklapper) [19:51:47] 10Gerrit (Gerrit 3.8), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886#9671097 (10hashar) > I have an issue with the Gerrit 3.8.4 release, it looks like it got tagged on Feb 23 and is missing a bunch of commits that... [19:53:19] 10Gerrit (Gerrit 3.8), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886#9671106 (10hashar) Regarding scheduling, next week has an holiday on Monday and the MediaWiki train is rolled during European morning. The Gerri... [19:55:59] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations, 10observability, and 2 others: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations' - https://phabricator.wikimedia.org/T360595#9671131 (10Ladsgroup) >>! In T360595#9665327, @colewhite wrote: >>>!... [20:02:31] thcipriani: what I said about "puppet 7" and distro upgrade was not true. True is "can not go to puppet 7 while still on buster" [20:02:43] this is also why they werent done yet but the test VM was [20:02:53] so distro first [20:13:52] 10Deployments, 06Release-Engineering-Team, 06serviceops, 13Patch-For-Review: 14httpbb appserver test breaks deployment of the week due to a timeout parsing page - 14https://phabricator.wikimedia.org/T360867#9671260 (10hashar) 14Great, thank you for the patches! -:] [20:20:50] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.42.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T360156#9671317 (10Nikerabbit) >>! In T360156#9670036, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations),... [20:21:17] (03CR) 10Hashar: [C:03+2] doc: add CodeMirror to opensource.yaml [integration/docroot] - 10https://gerrit.wikimedia.org/r/1015209 (https://phabricator.wikimedia.org/T359986) (owner: 10MusikAnimal) [20:21:53] (03Merged) 10jenkins-bot: doc: add CodeMirror to opensource.yaml [integration/docroot] - 10https://gerrit.wikimedia.org/r/1015209 (https://phabricator.wikimedia.org/T359986) (owner: 10MusikAnimal) [20:26:13] (03CR) 10Hashar: [C:03+2] zuul: Enable CI for mediawiki/gadgets/RTRC [integration/config] - 10https://gerrit.wikimedia.org/r/1015361 (owner: 10Krinkle) [20:27:26] (03Merged) 10jenkins-bot: zuul: Enable CI for mediawiki/gadgets/RTRC [integration/config] - 10https://gerrit.wikimedia.org/r/1015361 (owner: 10Krinkle) [20:28:01] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/1015361 [20:28:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:28:57] (03CR) 10Hashar: [C:03+2] "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1015360 (owner: 10Krinkle) [20:30:09] (03Merged) 10jenkins-bot: jjb: document mediawiki-core-doxygen-docker cronjob and task [integration/config] - 10https://gerrit.wikimedia.org/r/1015360 (owner: 10Krinkle) [20:30:15] James_F: if you are around MathSearch doesn't pass php 8.2 either :) [20:30:21] https://gerrit.wikimedia.org/r/c/integration/config/+/1015079 [20:30:38] I am deploying it [20:30:41] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/MathSearch] Disable PHP 8.2 testing for now [integration/config] - 10https://gerrit.wikimedia.org/r/1015079 (https://phabricator.wikimedia.org/T360709) (owner: 10Physikerwelt) [20:31:52] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/MathSearch] Disable PHP 8.2 testing for now [integration/config] - 10https://gerrit.wikimedia.org/r/1015079 (https://phabricator.wikimedia.org/T360709) (owner: 10Physikerwelt) [20:32:32] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/1015079 [20:32:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:54] mutante: thank you for the explanation about contint and Puppet 7 :) [20:33:56] hashar: ACK, np [20:34:30] * hashar sleeps [20:34:31] :) [20:41:24] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations, 10observability, and 2 others: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations' - https://phabricator.wikimedia.org/T360595#9671446 (10Ladsgroup) I made progress, will write down more once I'm... [20:52:51] hashar: thx for deploying the RTRC job :) [21:10:11] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 06collaboration-services: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9671533 (10Dzahn) The new deployment calender will be generated tomorrow. Then we should see our new GitLab window. [21:10:30] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 06collaboration-services: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470#9671534 (10Dzahn) 05Open→03In progress [22:24:13] 10GitLab-Application-Security-Pipeline, 13Patch-For-Review, 07Security: Design AppSec Pipeline metrics approach - https://phabricator.wikimedia.org/T342467#9671648 (10sbassett) Should be ready for review now: https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates-stats/-/merge_requests/1#no... [22:29:55] 10GitLab-Application-Security-Pipeline, 06Security-Team, 07SecTeam-Processed, 07Security: Update the image used by the python-bandit appsec include - https://phabricator.wikimedia.org/T360721#9671655 (10sbassett) Via the new appsec pipeline metrics cli (T342467), I was able to find that only a couple of re... [22:37:12] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9671679 (10brennen) a:05dduvall→03brennen > This seems to have tra... [22:37:14] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9671683 (10brennen) 05Open→03In progress [23:52:10] 10Continuous-Integration-Config, 10MediaWiki-Documentation, 10MediaWiki-Platform-Team (Radar), 05MW-1.39-notes, and 4 others: 14MediaWiki core docs unavailable for MW 1.35 and later - 14https://phabricator.wikimedia.org/T317451#9671956 (10apaskulin) 14Thanks for all your work on this, @Krinkle!