[00:38:00] (03PS1) 10Ebernhardson: Bump spark version to 2.4.4 in tox-pyspark docker image to match current deployment [integration/config] - 10https://gerrit.wikimedia.org/r/612447 [00:39:23] (03CR) 10jerkins-bot: [V: 04-1] Bump spark version to 2.4.4 in tox-pyspark docker image to match current deployment [integration/config] - 10https://gerrit.wikimedia.org/r/612447 (owner: 10Ebernhardson) [00:40:56] (03PS2) 10Ebernhardson: Bump spark version to 2.4.4 in tox-pyspark image [integration/config] - 10https://gerrit.wikimedia.org/r/612447 [00:57:28] (03CR) 10Jeena Huneidi: "Well, I think what I said earlier is not exactly true, but regardless this wouldn't have worked with credentials variables, so I'm working" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/610922 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi) [04:28:08] 10VPS-project-codesearch, 10cloud-services-team (Kanban): Switch codesearch to codesearch.wmcloud.org - https://phabricator.wikimedia.org/T256206 (10Andrew) [04:54:54] (03PS3) 10Jforrester: Add VisualEditor and TemplateData to the MW tarball [tools/release] - 10https://gerrit.wikimedia.org/r/610260 (https://phabricator.wikimedia.org/T243258) [04:55:37] (03CR) 10Jforrester: [C: 03+2] Add VisualEditor and TemplateData to the MW tarball [tools/release] - 10https://gerrit.wikimedia.org/r/610260 (https://phabricator.wikimedia.org/T243258) (owner: 10Jforrester) [04:56:10] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Jdforrester-WMF) [04:56:17] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Jdforrester-WMF) [04:56:22] (03Merged) 10jenkins-bot: Add VisualEditor and TemplateData to the MW tarball [tools/release] - 10https://gerrit.wikimedia.org/r/610260 (https://phabricator.wikimedia.org/T243258) (owner: 10Jforrester) [04:58:21] (03PS2) 10Jforrester: parameter_functions: VisualEditor and TemplateData are now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/611271 (https://phabricator.wikimedia.org/T243258) [05:12:39] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Prune /vendor for REL1_35 - https://phabricator.wikimedia.org/T256379 (10Jdforrester-WMF) I was thinking of doing it immediately, but yes. [05:13:00] (03CR) 10Jforrester: [C: 03+2] parameter_functions: VisualEditor and TemplateData are now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/611271 (https://phabricator.wikimedia.org/T243258) (owner: 10Jforrester) [05:14:03] (03Merged) 10jenkins-bot: parameter_functions: VisualEditor and TemplateData are now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/611271 (https://phabricator.wikimedia.org/T243258) (owner: 10Jforrester) [05:17:55] !log Zuul: parameter_functions: VisualEditor and TemplateData are now tarballed (no-op) [05:17:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:23:02] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [05:33:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [05:39:40] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.36-release: Expand the set of bundled extensions and skins in MediaWiki 1.36 - https://phabricator.wikimedia.org/T246381 (10Akuckartz) [05:44:53] !log Created WMF-deployed REL1_35 branches with `./branch.py --branchpoint 7d04152db4f8ea9a459511bed8117101d9bb4602 --bundle base --core-version 1.35.0-beta --task T256376 REL1_35` [05:44:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:44:55] T256376: Branch REL1_35 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T256376 [05:54:13] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) [06:08:13] (03CR) 10Jforrester: Fix up branch.py so that it's suitable for wmf/ production branches (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (https://phabricator.wikimedia.org/T222829) (owner: 1020after4) [06:10:55] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) The command would be `./branch.py --branchpoint master --bundle '*' --core-version 1.35.0-beta --task T256376 REL1_35`... [06:14:39] (03PS1) 10Jforrester: make-release: Add back wildcard branch support [tools/release] - 10https://gerrit.wikimedia.org/r/612466 [06:14:49] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/612466 [06:21:25] (03CR) 1020after4: [C: 03+2] "whoops! that was an oversight to say the least. Thanks James!" [tools/release] - 10https://gerrit.wikimedia.org/r/612466 (owner: 10Jforrester) [06:22:09] (03Merged) 10jenkins-bot: make-release: Add back wildcard branch support [tools/release] - 10https://gerrit.wikimedia.org/r/612466 (owner: 10Jforrester) [06:23:48] 10MediaWiki-Releasing, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10MW-1.35-notes, 10MW-1.35-release, 10Patch-For-Review: Branch REL1_35 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T256376 (10Jdforrester-WMF) 05Open→03... [06:23:50] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Release MW 1.35 - https://phabricator.wikimedia.org/T256375 (10Jdforrester-WMF) [06:45:31] https://phabricator.wikimedia.org/T253851#6303340 [06:45:54] So that repo is wierd. What's going on there? [07:25:21] (03CR) 10Awight: "> Thank you Adam for triggering this! :]" [integration/config] - 10https://gerrit.wikimedia.org/r/589438 (https://phabricator.wikimedia.org/T235118) (owner: 10Awight) [07:27:53] 10Diffusion, 10Phabricator, 10Operations, 10Puppet: Diffussion (Phabricator) operations-puppet repo synchronization error - https://phabricator.wikimedia.org/T257895 (10jcrespo) [07:29:38] 10Diffusion, 10Phabricator, 10Operations, 10Puppet: Diffusion (Phabricator) operations-puppet repo synchronization error - https://phabricator.wikimedia.org/T257895 (10jcrespo) [07:35:52] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10jcrespo) reviewdb is just backed up, but otrs backup on the same instance has yet to finish to make... [08:01:45] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) >>! In T255715#6303435, @jcrespo wrote: > reviewdb is just backed up, but otrs backup on... [09:07:09] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Jdforrester-WMF) OK, testwikis look fine for now. [09:22:42] (03CR) 10Legoktm: make-release: Add back wildcard branch support (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/612466 (owner: 10Jforrester) [09:23:26] (03CR) 10Legoktm: make-release: Add back wildcard branch support (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/612466 (owner: 10Jforrester) [09:23:29] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) Instruction above based on the last time this was done (by Dan, per SAL on 2019-10-04), but `./branch.py REL1_35 --bun... [09:24:23] (03CR) 10Jforrester: make-release: Add back wildcard branch support (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/612466 (owner: 10Jforrester) [09:24:57] (03PS1) 10Legoktm: make-release: Fix branch wildcard support [tools/release] - 10https://gerrit.wikimedia.org/r/612518 [09:26:44] (03CR) 10Jforrester: [C: 03+2] make-release: Fix branch wildcard support [tools/release] - 10https://gerrit.wikimedia.org/r/612518 (owner: 10Legoktm) [09:27:44] (03Merged) 10jenkins-bot: make-release: Fix branch wildcard support [tools/release] - 10https://gerrit.wikimedia.org/r/612518 (owner: 10Legoktm) [09:34:12] !log Running `./branch.py REL1_35 --bundle '*' --branchpoint master --core-version 1.35.0` for T257892 [09:34:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:34:15] T257892: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 [09:34:30] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) Thanks to the wonderful @legoktm, this is now done. [09:34:36] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Branch REL1_35 for all extensions and skins - https://phabricator.wikimedia.org/T257892 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [09:34:38] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Release MW 1.35 - https://phabricator.wikimedia.org/T256375 (10Jdforrester-WMF) [09:36:58] so, thoughts on removing the entire Revision class today? [09:37:12] sounds scary [09:38:01] I think that needs to wait a few releases [09:38:48] how about just removing the parts that use $wgUser - that was the whole reason I wanted to hard deprecate it in the first place [09:39:25] See https://www.mediawiki.org/wiki/Stable_interface_policy#Removal [09:40:23] * DannyS712 is chuckling to themselves [09:40:32] The constructor is one of the parts that uses $wgUSer [09:43:29] LOL [09:43:31] you got me [09:45:58] see https://phabricator.wikimedia.org/T247143 - many of the methods are already completely unused. Should we wait until 1.37 to start removing them, or is 1.36 okay (not today, but this version) [10:03:09] if something is clearly unused, I think it's probably OK, but I also don't see a significant harm in waiting till 1.37 [10:42:07] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO, 10Quality-and-Test-Engineering-Team (QTE), 10User-Soham, and 2 others: Create MediaWiki-Docker documentation for all repositories with Selenium tests - https://phabricator.wikimedia.org/T256239 (10Vidhi-Mody) [11:50:40] (03PS1) 10Tchanders: Zuul: [mediawiki/extensions/CheckUser] Add GuidedTour to phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/612554 [12:00:41] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10jcrespo) Bacula is running now. [12:03:01] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/CheckUser] Add GuidedTour to phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/612554 (owner: 10Tchanders) [12:04:06] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add GuidedTour to phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/612554 (owner: 10Tchanders) [12:10:17] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Jdforrester-WMF) Deployed to group0. One blocker identified (to do with Vecto... [12:25:41] (03PS1) 10Lucas Werkmeister (WMDE): layout: [mediawiki/extensions/Wikibase] make apitests voting [integration/config] - 10https://gerrit.wikimedia.org/r/612562 (https://phabricator.wikimedia.org/T257549) [12:27:08] (03PS2) 10Lucas Werkmeister (WMDE): layout: [mediawiki/extensions/Wikibase] make apitests voting [integration/config] - 10https://gerrit.wikimedia.org/r/612562 (https://phabricator.wikimedia.org/T257549) [12:28:12] (03CR) 10Lucas Werkmeister (WMDE): layout: [mediawiki/extensions/Wikibase] make apitests voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/612562 (https://phabricator.wikimedia.org/T257549) (owner: 10Lucas Werkmeister (WMDE)) [13:27:42] James_F: was 1.35 cut? [13:28:02] I'm looking at a +2'ing patch that proposes to add a note to release notes for 1.35 [13:28:22] yes, but can be backported [13:28:31] I'm not james though [13:28:40] k [13:29:29] should https://gerrit.wikimedia.org/r/c/mediawiki/core/+/609517 be backported as well? [13:35:13] kostajh: Yes. [13:35:59] DannyS712: It could be, if it's going to matter to third parties, but it looks like it's a pretty Wikimedia-specific concern? [13:36:58] I wonder if we should start a gerrit tag for backport candidates etc [13:37:22] DannyS712: yeah, I was thinking about it [13:37:23] Some are probably more obvious but some willl be subjective [13:38:09] idk. They're right that EditPage is a mess - hopefully I'll get to clean it up in 1.36 as part of T157658 [13:38:09] T157658: Factor out a backend from EditPage - https://phabricator.wikimedia.org/T157658 [13:38:29] if you want a sneak peak at my work - https://gerrit.wikimedia.org/r/c/mediawiki/core/+/600426 [13:38:51] don't worry, the patch is only around 6.3k lines [13:39:52] who are you going to trick into reviewing it? :P [13:40:37] I've tried to convince a couple people to owe me a review (I'll review these 10 patches of yours, but can you agree to review one of mine later?) but no luck so far [13:41:17] That works fine for smaller or simple patches... [13:41:31] did you mention which patch they would have to review? [13:41:35] heh [13:41:42] But for 6.3K that will almost certainly break prod due to edge cases and such... [13:41:53] because 10 to 1 seems like it has something fishy going on [13:42:02] lol I'm definitely going to split it up, the patch is just to see that it will all be able to come together successfully. see https://phabricator.wikimedia.org/T157658#6191234 for a roadmap [13:42:20] its like with the hooks at first - add the runner, and then individually migrate constraints [13:51:17] DannyS712: I'd recommend introducing the EditManager concept as an additional mechanism, uncalled except e.g. by a temporary API entry point, and then switch things over to it, as a way of moving forward. [13:53:55] hmm, the EditManager was going to be the backend that was factored out. But it'll definitely be split up - each of the constraints has 100% test coverage, which in some cases takes twice as many lines of code as the underlying constraint itself, and the addition of the EditConflictManager can be separate [14:02:08] nods. [14:02:25] It'd be nice to have some proper test coverage there. [14:05:45] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10jcrespo) a:05jcrespo→03Marostegui `lines=10,name=db backup metadata root@db2093.codfw.wmnet[zarc... [14:06:48] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10jcrespo) Last note: after db drop, grants for 'dump' users should be dropped too, so no backups are... [14:40:29] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10CPT Initiatives (API Gateway), 10MW-1.35-notes (1.35.0-wmf.40; 2020-07-07), and 5 others: New Public Wiki for the API Portal - https://phabricator.wikimedia.org/T246945 (10Urbanecm) [14:45:50] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Add tests for Scap cdb generation - https://phabricator.wikimedia.org/T254000 (10LarsWirzenius) [14:47:04] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Make a proof of concept set of nested VMs for train dev/test - https://phabricator.wikimedia.org/T253997 (10LarsWirzenius) 05Open→03Declined https://phabricator.wikimedia.org/T255695 is a duplicate. Clo... [14:49:54] 10Release-Engineering-Team (Deployment services), 10Scap: Experiment rsync "new compression" - https://phabricator.wikimedia.org/T252540 (10LarsWirzenius) 05Open→03Resolved This is fixed in the code since a while ago. Closing task. [14:53:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10LarsWirzenius) [14:53:38] 10Scap, 10Operations, 10serviceops: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10LarsWirzenius) 05Open→03Resolved --canary-wait-time has been included in a release and announced to the public and used on multiple trains now. Closing task. [14:56:45] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Ammarpad) [14:58:35] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Jdforrester-WMF) All now nominal. [14:59:42] Typically i add antoine to integration/config dockerfiles patches, checking the last few Change-Id's for that directory it seems antoine typically merges. Who else can I add to review since he's on vacation this week? [15:03:01] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10CPT Initiatives (API Gateway), 10MW-1.35-notes (1.35.0-wmf.40; 2020-07-07), and 5 others: New Public Wiki for the API Portal - https://phabricator.wikimedia.org/T246945 (10Reedy) @Urbanecm Why did you change the dbname? It's not going to be apiwikimedia [15:05:58] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10CPT Initiatives (API Gateway), 10MW-1.35-notes (1.35.0-wmf.40; 2020-07-07), and 5 others: New Public Wiki for the API Portal - https://phabricator.wikimedia.org/T246945 (10Urbanecm) @Reedy Hi, that was actually an automated change using https://githu... [15:10:57] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Isaac) @Reedy correct me if I'm wrong -- in practice, this would not noticeably change anything about our process of pushing changes to the research page? It... [15:12:14] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10CPT Initiatives (API Gateway), 10MW-1.35-notes (1.35.0-wmf.40; 2020-07-07), and 5 others: New Public Wiki for the API Portal - https://phabricator.wikimedia.org/T246945 (10Ladsgroup) >>! In T246945#6304696, @Urbanecm wrote: > @Reedy Hi, that was actu... [15:14:14] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [15:19:04] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) It'd basically be similar to any other repo. Changes may fail due to other updates being needed (such as out of date/node packages with known issues i... [15:19:59] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) I note, after adding the CI stuff, some remedial work might be needed to get things to a better state before moving forward; whether fixing issues or... [15:21:59] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [15:24:15] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [15:25:36] (03PS3) 10Jforrester: Bump spark version to 2.4.4 in tox-pyspark image [integration/config] - 10https://gerrit.wikimedia.org/r/612447 (owner: 10Ebernhardson) [15:25:41] (03CR) 10Jforrester: [C: 03+2] Bump spark version to 2.4.4 in tox-pyspark image [integration/config] - 10https://gerrit.wikimedia.org/r/612447 (owner: 10Ebernhardson) [15:26:27] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Isaac) > I note, after adding the CI stuff, some remedial work might be needed to get things to a better state before moving forward; whether fixing issues o... [15:27:59] (03Merged) 10jenkins-bot: Bump spark version to 2.4.4 in tox-pyspark image [integration/config] - 10https://gerrit.wikimedia.org/r/612447 (owner: 10Ebernhardson) [15:28:49] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) I haven't tried to run it locally, so don't know if/what is broken. But in theory, we can disable broken tests if there's an issue Only one way to fi... [15:37:43] !log Docker: Publishing tox-pyspark 0.6.0 [15:38:10] !log Docker: Publishing tox-pyspark 0.6.0 [15:38:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:38:25] * ebernhardson apparently tried to pull from docker 30s too early :) [15:38:39] I'll prep a jjb patch to bump the versions in a sec [15:56:39] (03PS1) 10Jforrester: jjb: Switch to latest tox-pyspark [integration/config] - 10https://gerrit.wikimedia.org/r/612623 [15:57:45] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/612623 (owner: 10Jforrester) [15:59:19] (03Merged) 10jenkins-bot: jjb: Switch to latest tox-pyspark [integration/config] - 10https://gerrit.wikimedia.org/r/612623 (owner: 10Jforrester) [16:37:20] 10Project-Admins: Project creation for tool - https://phabricator.wikimedia.org/T257937 (10RobH) [16:38:40] 10Release-Engineering-Team, 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10Krinkle) [16:38:50] 10Release-Engineering-Team, 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10Krinkle) [16:40:47] 10Release-Engineering-Team-TODO, 10observability, 10Developer Productivity: Add mwversion to php7-fatal-error.php logstash message - https://phabricator.wikimedia.org/T253781 (10Krinkle) [16:41:19] 10Release-Engineering-Team-TODO, 10observability, 10Developer Productivity: Add mwversion to php7-fatal-error.php logstash message - https://phabricator.wikimedia.org/T253781 (10Krinkle) p:05Triage→03Low a:03Krinkle [16:50:28] ebernhardson: Already done. [16:50:28] !log Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/611457 into deployment-puppetmaster04:/var/lib/git/operations/puppet [16:50:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:55:39] James_F: thanks! [16:56:38] 10Release-Engineering-Team, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10mpopov) [17:05:28] (03PS2) 10C. Scott Ananian: Turn on "integrated" phan tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612429 (https://phabricator.wikimedia.org/T248726) [17:05:30] (03PS2) 10C. Scott Ananian: Turn on standard quibble tests for Parsoid as an extension [integration/config] - 10https://gerrit.wikimedia.org/r/612430 (https://phabricator.wikimedia.org/T227352) [17:07:56] (03CR) 10jerkins-bot: [V: 04-1] Turn on "integrated" phan tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612429 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [17:07:59] (03CR) 10jerkins-bot: [V: 04-1] Turn on standard quibble tests for Parsoid as an extension [integration/config] - 10https://gerrit.wikimedia.org/r/612430 (https://phabricator.wikimedia.org/T227352) (owner: 10C. Scott Ananian) [17:10:29] hey releng. is there a gerrit group the person needs to belong to to be able to do 'recheck'? [17:10:46] here https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaModeration/+/579392/14#message-743e9a5ff82cf30077fa3798171081193d30a84e it didn't work for the patch author, but works for me [17:11:43] Pchelolo: Need to be CI whitelisted [17:12:31] Reedy: how do you add a person to it? [17:12:43] I don't see it in groups.. [17:12:49] integration/config zuul/layout.yaml [17:13:01] https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L73 [17:13:29] oh! gotcha, thank you! will update [17:13:42] Reedy: allowlisted, not whitelisted [17:14:12] Yes. Unfortunately old habits die hard [17:14:36] also, allowlisted sounds stupid [17:14:36] I have the exact same problem, "allowlist" somehow sounds weird to me [17:14:57] "they have to be on the approved list" [17:15:42] I mean, they just need to on the list [17:15:52] It's not like we have a block list too (in this case) [17:53:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10Patch-For-Review: beta-publish-deb failure do to "/srv/packages/db/LOG: permission denied" - https://phabricator.wikimedia.org/T250157 (10dancy) I cherry-picked https://... [18:42:34] allowedlist is ok [18:42:48] whatever, naming is hard [18:54:14] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [19:04:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [19:31:05] (03PS3) 10C. Scott Ananian: Turn on "integrated" phan tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612429 (https://phabricator.wikimedia.org/T248726) [19:31:07] (03PS3) 10C. Scott Ananian: Turn on standard quibble tests for Parsoid as an extension [integration/config] - 10https://gerrit.wikimedia.org/r/612430 (https://phabricator.wikimedia.org/T227352) [19:34:17] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10MediaWiki-Debug-Logger, 10Performance-Team, and 3 others: Ensure flood of hard-deprecations are caught during (train) deployments - https://phabricator.wikimedia.org/T252923 (10Krinkle) [19:41:20] (03CR) 10Reedy: [C: 03+2] Turn on "integrated" phan tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612429 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [19:42:20] (03Merged) 10jenkins-bot: Turn on "integrated" phan tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612429 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [19:43:15] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612429 [19:43:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:54:44] (03CR) 10Reedy: [C: 03+2] Turn on standard quibble tests for Parsoid as an extension [integration/config] - 10https://gerrit.wikimedia.org/r/612430 (https://phabricator.wikimedia.org/T227352) (owner: 10C. Scott Ananian) [19:55:39] (03Merged) 10jenkins-bot: Turn on standard quibble tests for Parsoid as an extension [integration/config] - 10https://gerrit.wikimedia.org/r/612430 (https://phabricator.wikimedia.org/T227352) (owner: 10C. Scott Ananian) [19:56:33] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612430 [19:56:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:17:46] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Reedy) [20:30:15] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Count_Count) This should really have **Unbreak now!** priority as users/visitors perceive it as a security issue. [20:31:16] Krenair: ^ If you're bored, looks like the LE cert didn't renew properly [20:33:24] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Krenair) p:05Triage→03Unbreak! a:03Krenair looking [20:35:59] Reedy, it's interesting, there is a newer cert on the box, guess it's not being served for some reason [20:36:02] puppet can't run because OOM [20:36:18] niiiiice [20:38:37] 10Release-Engineering-Team (Logspam), 10Commons, 10MediaWiki-File-management, 10Patch-For-Review, 10Wikimedia-production-error: PHP Warning: Invalid argument supplied for foreach() in FormatMetadata - https://phabricator.wikimedia.org/T257497 (10Umherirrender) 05Open→03Resolved p:05Triage→03Medium [20:40:49] Heads up: https://en.wikipedia.beta.wmflabs.org/wiki/ certificate expired 40 minutes ago [20:41:40] read up [20:42:15] no log [20:42:51] 10Beta-Cluster-Infrastructure, 10Core Platform Team: Beta cluster changeprop job queue down since 2020-06-18 - https://phabricator.wikimedia.org/T257620 (10BPirkle) p:05Triage→03Medium [20:43:08] I guessed someone wrote it already, so just to be sure it's known. [20:44:51] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Krenair) it's UBN because beta is down and this task is the beta project, not due to perceived security risk (it's only beta) initial glance: certs on the box look fine: ` root@deployment-... [20:45:18] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO, 10Wikibugs: wikibugs2 should not post on IRC for Gerrit changes marked 'WIP' - https://phabricator.wikimedia.org/T239928 (10dancy) 05Open→03Resolved [20:50:54] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Krenair) p:05Unbreak!→03High the immediate problem is solved by me manually doing the cert reload [21:01:32] maybe all the acme-chief related puppet errors made puppet skip the reload exec [21:02:09] uwsgi-acme-chief logs on -acme-chief03 show calls like `/puppet/v3/file_content/acmedata/mx/bfcd4752e6b346289533bcb6934671a2/rsa-2048.crt.key?environment=production&` [21:02:37] errors in puppet say Error 400 on SERVER: part must be in ['ec-prime256v1.crt', 'ec-prime256v1.chain.crt', 'ec-prime256v1.chained.crt', 'ec-prime256v1.key', 'ec-prime256v1.ocsp', 'rsa-2048.crt', 'rsa-2048.chain.crt', 'rsa-2048.chained.crt', 'rsa-2048.key', 'rsa-2048.ocsp'] [21:03:28] i.e. no .crt.key combined CERTIFICATE_TYPE [21:04:08] which is interesting because that feature arrived in v0.26 on 15th June, which this box thinks it has [21:04:27] oh but I guess we need to restart uwsgi-acme-chief :/ [21:04:36] Active: active (running) since Thu 2020-04-23 03:17:02 UTC; 2 months 21 days ago [21:05:32] there we go that's better [21:05:34] Notice: /Stage[main]/Profile::Trafficserver::Tls/Profile::Trafficserver::Tls_material[unified]/Acme_chief::Cert[unified]/File[/etc/acmecerts/unified/bf5e612e77174be08859dcf617306e92/ec-prime256v1.crt.key]/ensure: defined content [21:05:41] Notice: /Stage[main]/Profile::Trafficserver::Tls/Profile::Trafficserver::Tls_material[unified]/Acme_chief::Cert[unified]/File[/etc/acmecerts/unified/bf5e612e77174be08859dcf617306e92/rsa-2048.crt.key]/ensure: defined content [21:06:29] wonder if Package['acme-chief'] should notify Service['uwsgi-acme-chief'] on the acme-chief instance... [21:06:57] puppet on the client is looking better but is still upset about not having enough memory to do some varnishkafka/traffic-pool things [21:14:17] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.79 ms [21:15:06] restarted trafficserver service in -cache-text06, that freed enough memory to run puppet [21:15:48] less likely to occur in prod I guess as it'll have a lot more RAM to work with, beta's frontend only has 4GB :) [21:17:21] upload looks happy [21:23:31] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Krenair) @vgutierrez: I'm guessing puppet had failed to run the reload exec itself due to the errors connecting to acme-chief (`Error 400 on SERVER: part must be in ['ec-prime256v1.crt', '... [21:24:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [21:26:53] * Krenair disappears in a puff of smoke [21:33:29] what color though [22:27:12] (03PS1) 10Arlolra: Turn on API integration tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612673 (https://phabricator.wikimedia.org/T233736) [23:05:38] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Consider what paging who should get for what in RelEng/EngProd? - https://phabricator.wikimedia.org/T247427 (10thcipriani) [23:06:12] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Consider what paging who should get for what in RelEng/EngProd? - https://phabricator.wikimedia.org/T247427 (10thcipriani) 05Open→03Resolved Calling this resolved as its decided...now have to do it. [23:16:52] 10MediaWiki-Releasing, 10Core Platform Team, 10MediaWiki-Vendor, 10MW-1.35-notes, 10MW-1.35-release: Prune /vendor for REL1_35 - https://phabricator.wikimedia.org/T256379 (10Legoktm) a:03Legoktm [23:31:30] 10Project-Admins: Project creation for tool - https://phabricator.wikimedia.org/T257937 (10DannyS712) p:05Triage→03Low [23:54:19] PROBLEM - Host deployment-sentry01 is DOWN: CRITICAL - Host Unreachable (172.16.5.16) [23:54:36] (03CR) 10Krinkle: Add UnaryMinusSpacing sniff to remove spaces after unary minus (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591425 (https://phabricator.wikimedia.org/T155627) (owner: 10DannyS712) [23:57:05] (03PS4) 10DannyS712: Add UnaryMinusSpacing sniff to remove spaces after unary minus [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591425 (https://phabricator.wikimedia.org/T155627) [23:57:09] (03CR) 10DannyS712: Add UnaryMinusSpacing sniff to remove spaces after unary minus (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591425 (https://phabricator.wikimedia.org/T155627) (owner: 10DannyS712)