[05:28:37] are we still trying to deploy .28 or will we just skip to .29? [08:04:12] 10Project-Admins, 10Security, 10user-sbassett: Create security-related project tags secteam-reviewed and wikimedia-project-event - https://phabricator.wikimedia.org/T271729 (10Aklapper) @sbassett: Hmm, why were Edit Policy and Join Policy of #SecTeam-Processed and #SecTeam-wikimedia-project-event changed to... [08:11:50] Yippee, build fixed! [08:11:51] Project mwcore-phpunit-coverage-master build #1190: 09FIXED in 5 hr 11 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1190/ [08:20:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Joe) >>! In T266055#6784070, @thcipriani wrote: > > >>>! I... [08:27:40] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10Aklapper) This open task is only tagged with the RelEng Q2 project tag which recently has been archived. Please add an active project tag... [08:27:43] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Write script to merge key+signature files into single files in pgp-public-keys repository - https://phabricator.wikimedia.org/T258271 (10Aklapper) This open task is only tagged with the RelEng Q2 project tag which recently has been archived. Please... [08:27:47] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): echostore helm test service checker failing in staging cluster - https://phabricator.wikimedia.org/T259686 (10Aklapper) This open task is only tagged with the RelEng Q2 project tag which recently has been archived. Please add an active project tag... [08:27:52] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Hosting wikibase release tarballs on releases.wikimedia.org - https://phabricator.wikimedia.org/T268345 (10Aklapper) This open task is only tagged with the RelEng Q2 project tag which recently has been archived. Please add an active project tag to... [08:27:55] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Change deploy-promote to check previous groups have been promoted to - https://phabricator.wikimedia.org/T269242 (10Aklapper) This open task is only tagged with the RelEng Q2 project tag which recently has been archived. Please add an active projec... [08:32:54] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Hosting wikibase release tarballs on releases.wikimedia.org - https://phabricator.wikimedia.org/T268345 (10WMDE-leszek) 05Open→03Resolved a:03WMDE-leszek I think we're done here. Thanks RelEng for great help, and apologies for not managing ou... [09:15:51] !log devtools: switched gerrit-prod-1001 to local puppetmaster [09:15:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:31:01] (03PS1) 10JMeybohm: Use the UID instead of username in USER instructions [blubber] - 10https://gerrit.wikimedia.org/r/660771 (https://phabricator.wikimedia.org/T228967) [09:43:22] (03PS1) 10Hashar: scap: config for devtools on WMCS [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/660773 [09:45:33] (03CR) 10Hashar: [C: 04-1] "Cant figure out the ssh config so ... not much to do ;)" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/660773 (owner: 10Hashar) [09:49:22] (03Abandoned) 10Hashar: scap: config for devtools on WMCS [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/660773 (owner: 10Hashar) [10:18:19] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10hashar) Go blocked due to the introduction of {T254838} and User::changeable(By)Groups bein... [10:55:18] !log deployment-prep: Create beta votewiki (T272608) [10:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:55:21] T272608: SecurePoll: Add "electcomm" and "electionadmin" groups to beta - https://phabricator.wikimedia.org/T272608 [10:57:46] 10Release-Engineering-Team-TODO: Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10LarsWirzenius) [10:58:04] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Write script to merge key+signature files into single files in pgp-public-keys repository - https://phabricator.wikimedia.org/T258271 (10LarsWirzenius) 05Open→03Declined [10:58:26] 10Release-Engineering-Team-TODO: Change deploy-promote to check previous groups have been promoted to - https://phabricator.wikimedia.org/T269242 (10LarsWirzenius) [11:08:38] 10Release-Engineering-Team, 10Wikimedia-Site-requests: test2wiki is in group1 rather than group0 - https://phabricator.wikimedia.org/T273435 (10Urbanecm) [11:21:39] Project beta-update-databases-eqiad build #48077: 04FAILURE in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/48077/ [11:26:42] 10Diffusion: Diffusion support for viewing the raw file in the browser - https://phabricator.wikimedia.org/T117621 (10Aklapper) [11:26:50] 10Diffusion, 10Phabricator, 10Security: Diffusion 'view raw file' policy and behavior is unclear and inconsistent - https://phabricator.wikimedia.org/T149152 (10Aklapper) 05Stalled→03Open p:05Medium→03Low Still valid: Clicking "Raw file" in Diffusion's file view shows a raw file for php; doing that f... [11:27:06] 10Diffusion, 10Phabricator, 10Security: Diffusion 'view raw file' policy and behavior is unclear and inconsistent (PHP files displayed in browser; JS files offer downloading file) - https://phabricator.wikimedia.org/T149152 (10Aklapper) [11:46:03] 10Beta-Cluster-Infrastructure, 10User-zeljkofilipin, 10WorkType-NewFunctionality: Make selenium users use botflags at beta-cluster by setting the bot flag when saving edits - https://phabricator.wikimedia.org/T116027 (10Aklapper) [11:48:03] 10Beta-Cluster-Infrastructure, 10User-zeljkofilipin, 10WorkType-NewFunctionality: Make selenium users use botflags at beta-cluster by setting the bot flag when saving edits - https://phabricator.wikimedia.org/T116027 (10Majavah) I don't think that the bot flag can be used when doing edits from the browser. [11:57:51] 10Beta-Cluster-Infrastructure, 10User-zeljkofilipin, 10WorkType-NewFunctionality: Make selenium users use botflags at beta-cluster by setting the bot flag when saving edits - https://phabricator.wikimedia.org/T116027 (10Aklapper) On a related note, I found https://lists.wikimedia.org/pipermail/wikitech-l/201... [12:21:42] Yippee, build fixed! [12:21:42] Project beta-update-databases-eqiad build #48078: 09FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/48078/ [12:26:07] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Performance-Team (Radar): Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640 (10Urbanecm) [13:07:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap: scap: look at removing scap/sh.py - https://phabricator.wikimedia.org/T222372 (10LarsWirzenius) 05Open→03Resolved [13:33:03] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10Urbanecm) [13:40:59] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10Majavah) [13:41:04] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10Majavah) [14:42:20] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10hashar) 1.36.0-wmf.28 has reached group 1 an hour or so ago. Thanks @Urbanecm for the backp... [14:49:42] 10Release-Engineering-Team, 10Wikimedia-Site-requests: test2wiki is in group1 rather than group0 - https://phabricator.wikimedia.org/T273435 (10thcipriani) 05Open→03Invalid This was done circa 2017 in {T182326} to cover the use case of testing testwikidata with the next version. So testwiki and testwikidat... [14:57:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10thcipriani) One of the things I'm hopeful about after implem... [16:04:36] !log Cherry-picking https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660063 in beta. [16:04:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:05:29] 10Project-Admins, 10Security, 10user-sbassett: Create security-related project tags secteam-reviewed and wikimedia-project-event - https://phabricator.wikimedia.org/T271729 (10sbassett) >>! In T271729#6790352, @Aklapper wrote: > @sbassett: Hmm, why were Edit Policy and Join Policy of #SecTeam-Processed and #... [16:28:02] 10MediaWiki-Codesniffer: `Function name "onRecentChange_save" should use lower camel case` for inherited methods - https://phabricator.wikimedia.org/T273482 (10Daimona) [16:46:44] 10Release-Engineering-Team-TODO, 10WVUI, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), 10Upstream, 10Vue.js: Modify readme.md for gerrit markdown - https://phabricator.wikimedia.org/T269300 (10Volker_E) p:05Triage→03Medium [18:05:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 3 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) Here's the list of most of the stuff th... [18:19:47] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Pchelolo) [18:40:55] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10SDAW-MediaSearch (MediaSearch-ReleaseCandidate), 10Structured-Data-Backlog (Current Work), 10Wikimedia-extension-review-queue: Split MediaSearch out into its own extension - https://phabricator.wikimedia.org/T265939 (10AnneT) [19:00:23] (03CR) 10Dduvall: [C: 04-1] "Thanks for the patch! I just left one bit of feedback about getting rid on the `Name` field in `build.User`. Barring that I'm good for a C" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/660771 (https://phabricator.wikimedia.org/T228967) (owner: 10JMeybohm) [19:10:23] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) [19:23:43] 10Continuous-Integration-Config, 10API-Portal, 10MediaWiki-extensions-WikimediaApiPortalOAuth: Disable unit tests in CI for WikimediaApiPortalOAuth extension - https://phabricator.wikimedia.org/T273326 (10apaskulin) Thanks, @hashar! This is good to know. So is it ok to ignore the [post-merge build failure](h... [19:30:31] (03CR) 10Dduvall: [C: 03+2] "Looks good!" [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [19:31:40] (03CR) 10jerkins-bot: [V: 04-1] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [19:35:29] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Sustainability (Incident Followup), 10User-brennen: Add a quick rollback command to scap that skips all unnecessary steps - https://phabricator.wikimedia.org/T244544 (10thcipriani) 05Open→03Resolved a:05mmodell→03dancy I bel... [19:36:44] 10Release-Engineering-Team-TODO, 10User-greg, 10Wikimedia-extension-review-queue: Investigate and make improvements to the extension review process - https://phabricator.wikimedia.org/T195244 (10thcipriani) [19:40:23] (03CR) 10Dduvall: [C: 04-1] "See comment about the need for a deterministic map iteration. The test randomly caught it during gate-and-submit!" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [19:56:25] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10User-brennen: Applying security patches should be robust and also give some useful output - https://phabricator.wikimedia.org/T269153 (10brennen) This might warrant a couple of sepa... [20:10:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 3 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) >>! In T273312#6788663, @Legoktm wrote:... [20:46:22] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10dancy) [20:50:32] (03CR) 10Legoktm: [V: 03+2 C: 03+2] Allow LabelBots to vote V[+-]1 [extensions/NavigationTiming] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/660014 (owner: 10DannyS712) [20:53:50] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) [21:00:05] Project mwcore-phpunit-coverage-master build #1191: 04FAILURE in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1191/ [21:00:46] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10dancy) [21:07:11] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) @hasher et al, after discussion in IRC we release engineering has settled on rolling... [21:23:42] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) 05Open→03Declined [21:29:54] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) Rollback to .27 has finished. [22:23:23] (03PS8) 10Jeena Huneidi: apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) [22:37:28] wtf my ip range is globally blocked? this is strange... [22:38:02] The block was made by Vituzzu (meta.wikimedia.org). The reason given is leaky colo + open proxy at 206.226.72.49. the blocked range is 206.226.64.0/18 [22:41:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) > * if the server is pooled and you cannot depool i... [22:41:36] thcipriani: curious if you think the same? ^ [22:43:03] twentyafterfour: did the block predate 2019-10-24 ? [22:43:48] Krinkle: the date says it started in 2016 but I would think that I've been within the blocked range before but maybe I got assigned a new ip range by my ISP and didn't notice [22:44:31] yeah, ip whois says this IP range was created for "Mediacom Communications Corporation" on that date [22:45:05] Krinkle: right that's my isp [22:45:30] Start of block: 21:37, 15 July 2016 Expiry of block: 21:37, 15 July 2021 [22:45:45] but no block id # is shown [22:47:40] seems plausible that someone in the range was running an open proxy in the past but that's a long term block for a fairly big chunk of IP address space which is used by end users so that seems like an excessive block IMO [22:51:02] meh, email sent to stewards@ [22:51:25] twentyafterfour: ugh, hope it's resolved quickly [22:51:49] * thcipriani reads task [22:51:51] greg-g: me too, I have an unsaved phabricator documentation page that I put a lot of effort into writing [22:52:13] guess I'll save the wikitext locally [22:56:15] twentyafterfour: yes, please do :) [22:58:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10thcipriani) >>! In T266055#6794263, @Krinkle wrote: >> * if... [23:00:20] Krinkle: good question/point as to whether php-fpm has a graceful mechanism -- my assumption was that it didn't and therefore, you're right: normal deploys would become emergency deploys in a potentially unpredictable way; which would not be ideal [23:01:52] (03CR) 10Jeena Huneidi: apt.go: Add ability to target releases (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:09:36] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10Urbanecm) [23:10:15] (03CR) 10Dduvall: [C: 03+2] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:14:25] (03CR) 10jerkins-bot: [V: 04-1] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:18:19] (03CR) 10Dduvall: [C: 03+2] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:20:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) > Differentiating a failed restart […] Maybe it ca... [23:22:23] (03CR) 10jerkins-bot: [V: 04-1] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:28:10] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10thcipriani) >>! In T266055#6794406, @Krinkle wrote: >> Diffe... [23:39:35] thcipriani: ok, I dont know what pool.map() does or what _/failed hold. [23:39:48] do we currently capture stderr and use emptiness to determinen failure? [23:39:57] that woudl mean we can't use stderr with success indeed [23:40:34] that seems doable, "just" needs implementing [23:40:48] might work if we use status code as failure signal with the custom message we have today, and if success then just forward stderr as warning [23:40:53] oh it is already always non-fatal [23:41:02] so we don't increment the failure counter/threshold [23:41:28] I guess that's fine then [23:41:35] so we don't need to distinguish them in that case [23:41:47] except maybe to improve the real failure error message so that we don't need a custom wrapper perhaps? [23:42:17] %d hosts [23:42:22] ah, this is an aggregate? [23:42:26] yep [23:42:28] er [23:42:30] lemme check [23:42:56] so `failed` is already some sort of mw123: sterr lines thingy maybe? [23:45:32] looks like we yield a tuple '(host, status)' and that's what gets aggregated [23:46:22] where status is exit code [23:46:51] but(!) I see that the ssh.job function warns on failures anyway [23:47:11] tl;dr: it seems that if this fails, the deployer will see stderr [23:49:19] (03CR) 10Jeena Huneidi: [C: 03+2] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [23:52:59] (03CR) 10jerkins-bot: [V: 04-1] apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi)