[00:26:29] Project beta-scap-eqiad build #277998: 04FAILURE in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/277998/ [00:36:28] Yippee, build fixed! [00:36:29] Project beta-scap-eqiad build #277999: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/277999/ [00:48:47] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Aklapper) Gerrit itself would be off-topic here, however: I don't think that some review-bot sending more mail solves a prob... [00:58:33] 10Phabricator: Shouldn't #good_first_bug be #good_first_task? - https://phabricator.wikimedia.org/T239221 (10Aklapper) 05Open→03Resolved Made a few edits on wiki pages which mention the name of this tag and not the name of that mw.org page... [03:25:45] 10Phabricator: Evaluate Phabricator for its ability to export and import data - https://phabricator.wikimedia.org/T108199 (10Weshouman) I have not tried Conduit API yet, it could be a step into importing and exporting, but it's just much relevant if there is some exporter/importer, even if it worked only with ph... [04:45:00] 10Phabricator: Evaluate Phabricator for its ability to export and import data - https://phabricator.wikimedia.org/T108199 (10Dzahn) @Weshouman I am not sure if that's what this is about but i want to point out there are Phabricator dumps at https://dumps.wikimedia.org/other/misc/ nowadays and the last comment o... [05:01:42] 10Phabricator: Evaluate Phabricator for its ability to export and import data - https://phabricator.wikimedia.org/T108199 (10ArielGlenn) The dumps referenced above are missing some content and have been a bit fidgety to maintain. For more on that, see T236507. [09:09:01] 10Phabricator, 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10User-Sebastian_Berlin-WMSE, 10Wikispeech-jobrunner (Sprint): Automatically add tasks in some Wikispeech projects to column of another Wikispeech project - https://phabricator.wikimedia.org/T239390 (10Sebastian_Berlin-WMSE) [09:09:24] 10Phabricator, 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10User-Sebastian_Berlin-WMSE, 10Wikispeech-jobrunner (Sprint): Automatically add tasks in some Wikispeech projects to column of another Wikispeech project - https://phabricator.wikimedia.org/T239390 (10Sebastian_Berlin-WMSE) [09:09:45] 10Phabricator, 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10User-Sebastian_Berlin-WMSE, 10Wikispeech-jobrunner (Sprint): Automatically add tasks in some Wikispeech projects to column of another Wikispeech project - https://phabricator.wikimedia.org/T239390 (10Sebastian_Berlin-WMSE) [10:34:32] !log Purged pbuilder / cowbuilder etc from integration-slave-jessie-1002 and integration-slave-jessie-1004 # T224943 [10:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:34:36] T224943: Migrate debian-glue jobs to Buster instances - https://phabricator.wikimedia.org/T224943 [10:35:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Patch-For-Review: Migrate debian-glue jobs to Buster instances - https://phabricator.wikimedia.org/T224943 (10hashar) 05Open→03Resolved [10:35:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Wikidata, and 3 others: Update Doxygen in CI to 1.8.15 or greater - https://phabricator.wikimedia.org/T239482 (10hashar) [11:11:40] 10Phabricator: Evaluate Phabricator for its ability to export and import data - https://phabricator.wikimedia.org/T108199 (10Aklapper) @Weshouman: Welcome to Wikimedia's Phabricator instance. If you have general comments about the Phabricator software (which are not specific about the instance of Phabricator run... [11:25:38] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10good first task: Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10hashar) [11:29:48] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912): Migrate / phaseout out CI job phabricator-jessie-diffs - https://phabricator.wikimedia.org/T239980 (10hashar) [11:33:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) [11:33:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) [11:33:52] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912): Migrate / phaseout out CI job phabricator-jessie-diffs - https://phabricator.wikimedia.org/T239980 (10hashar) [11:33:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10good first task: Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10hashar) [11:37:31] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) [11:37:42] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) p:05Triage→03Normal [11:37:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912): Migrate / phaseout out CI job phabricator-jessie-diffs - https://phabricator.wikimedia.org/T239980 (10hashar) p:05Triage→03Normal [11:38:00] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10good first task: Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10hashar) p:05Triage→03Normal [11:40:36] 10Release-Engineering-Team, 10serviceops: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Lucas_Werkmeister_WMDE) >>! In T238956#5716923, @mmodell wrote: > I see the "related gerrit patches" box on this very page. Yup, it’s back now. Thanks! [11:58:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Wikidata, and 3 others: Update Doxygen in CI to 1.8.15 or greater - https://phabricator.wikimedia.org/T239482 (10hashar) The Doxygen CI container is build using: {nav doxygen < p... [12:17:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Patch-For-Review: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 (10hashar) Packages for Stretch ( stretch-wikimedia/... [12:18:53] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Jenkins: Add latest jenkins debian packages to apt.wikimedia.org and upgrade jenkins to latest LTS (2.190.3) - https://phabricator.wikimedia.org/T239586 (10hashar) Thanks for the new Jenkins packages :] Fo... [12:20:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Jenkins: Upgrade Jenkins to 2.190.3 - https://phabricator.wikimedia.org/T239985 (10hashar) [12:52:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Jenkins: Upgrade Jenkins to 2.190.3 - https://phabricator.wikimedia.org/T239985 (10hashar) p:05Triage→03High [15:56:10] 20k media on beta commons with captions and depicts fot folks' testing pleasure [16:41:30] 10Phabricator: Do not direct /maniphest to security-team-needs-triage - https://phabricator.wikimedia.org/T240011 (10Apap04) [16:42:42] 10Phabricator: Do not direct /maniphest to security-team-needs-triage - https://phabricator.wikimedia.org/T240011 (10Apap04) p:05Triage→03Low [16:47:59] 10Phabricator: Do not direct /maniphest to security-team-needs-triage as default global query - https://phabricator.wikimedia.org/T240011 (10Aklapper) [16:48:36] 10Phabricator, 10Security-Team: Do not direct /maniphest to security-team-needs-triage as default global query - https://phabricator.wikimedia.org/T240011 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks for finding this! For future reference please follow https://www.mediawiki.org/wiki/How_to_report_a... [16:50:48] 10Phabricator, 10Security-Team: Do not direct /maniphest to security-team-needs-triage as default global query - https://phabricator.wikimedia.org/T240011 (10Apap04) 'Twas quick. Thanks @Aklapper! I'll report more bugs like these more clearly. [17:04:14] (03CR) 10Jforrester: "Used in Icc0a973a857e2dd9da0786e3fa84db975bf83379." [integration/quibble] - 10https://gerrit.wikimedia.org/r/547843 (https://phabricator.wikimedia.org/T192167) (owner: 10Jforrester) [17:15:43] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10good first task: Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10Jdforrester-WMF) Or we could just... [17:41:14] (03CR) 10Thcipriani: [C: 03+2] "changes to make wmf branch of mw/core locally look good to me. Other good housekeeping changes as well. Let's merge and iterate from this " [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (https://phabricator.wikimedia.org/T222829) (owner: 1020after4) [17:41:38] thcipriani: Hurrah. [17:41:46] (03Merged) 10jenkins-bot: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (https://phabricator.wikimedia.org/T222829) (owner: 1020after4) [17:51:47] ^ nice. [18:07:04] +1 [18:10:10] so, idea: instead of a bulk train branch cut all at once, let's automatically cherry-pick patches, one by one from master into production.git say every 5 minutes, then deploy exactly one commit at a time, then wait a few minutes... that way any anomaly in logs will be temporally correlated with ~1 commit [18:10:21] alternate idea: [18:12:31] nothing makes it into production.git until it gets a +2 from any one of a) team lead or manager in charge of the affected code, b) developer who wrote the patch, or c) any release engineer or otherwise qualified person (e.g. a SWAT deployer who is willing to be responsible for the patch). But as soon as that +2 happesn then the patch gets automatically cherry-picked to production.git and deployed [18:12:34] immediately [18:13:55] (by immediately I mean first deployed to debug / canary servers, then gradually rolled out across the cluster) [18:15:36] twentyafterfour: Deploys and their impacts on the logs take up to an hour to percolate through… [18:15:58] (03PS1) 10Dduvall: Commit complete go.mod and go.sum files [blubber] - 10https://gerrit.wikimedia.org/r/555562 [18:16:08] that's still narrows down the likely culprits to maybe 5 patches [18:16:13] (But other than that, isn't that what we're building with the containerisation work?) [18:16:51] hopefully? [18:17:58] I thought that the way we manage git branches and the way patches are integrated is still very much debatable and not really set in stone (do we have a consensus? in the past it's been very controversial [18:20:38] Oh. Certainly, I'm not planning to have wmf/ branches in the containerised world. [18:20:51] If someone thinks we should retain them, they haven't mentioned it. [18:22:40] I'm not sure when the world will be containerized proper but I think we should do something like the above sooner than later, with or without containers [18:23:17] Oh. [18:23:21] I disagree, a lot. [18:23:29] Moving to containers is a lot of work. [18:23:46] Moving to long-running branches is a lot of work. [18:24:12] I don't think it's sane to try to do the latter in the middle of the former, it'll just make them both rushed and badly done. [18:29:06] (03PS1) 10Dduvall: Fix `make clean` when project source is outside GOPATH [blubber] - 10https://gerrit.wikimedia.org/r/555568 [18:30:15] If the containerisation work is going to be paused for a year, maybe. [18:30:22] But it's unclear right now what the future is. [18:32:28] (03PS2) 10Dduvall: Fix `make clean` when project source is outside GOPATH [blubber] - 10https://gerrit.wikimedia.org/r/555568 [18:39:38] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MobileFrontend, 10Readers-Web-Backlog: Add curl support to mwext-node10-rundoc-docker - https://phabricator.wikimedia.org/T239246 (10hashar) **The overview** For MobileFrontend, CI does `npm run-script doc` w... [18:40:00] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MobileFrontend, 10Readers-Web-Backlog: Add curl support to mwext-node10-rundoc-docker - https://phabricator.wikimedia.org/T239246 (10hashar) p:05Triage→03Normal [18:41:56] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MobileFrontend, 10Readers-Web-Backlog: Add curl support to mwext-node10-rundoc-docker - https://phabricator.wikimedia.org/T239246 (10hashar) Note: quibble is the testrunner for mediawiki and has some fancy uti... [18:56:09] (03PS1) 10Dduvall: Exclude binaries from blubber images [blubber] - 10https://gerrit.wikimedia.org/r/555572 [18:56:57] (03PS1) 10Dduvall: Avoid extra image layers due to sparse user logs [blubber] - 10https://gerrit.wikimedia.org/r/555573 [18:58:13] (03CR) 10jerkins-bot: [V: 04-1] Avoid extra image layers due to sparse user logs [blubber] - 10https://gerrit.wikimedia.org/r/555573 (owner: 10Dduvall) [19:07:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Parsoid-PHP, 10Patch-For-Review: Installing composer modules for deployment - https://phabricator.wikimedia.org/T213494 (10ssastry) [19:22:29] James_F: I agree we shouldn't try to do one thing in the middle of the other. I just think that the way patches are deployed is sort of step 1 and doing the same thing inside containers is step 1.1 [20:18:31] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Dzahn) [20:21:00] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Dzahn) >>! In T236309#5678505, @Andrew wrote: > For the new project, please file a ticket here: https://phabricator.wikimedia.org/project/... [20:22:53] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Dzahn) 05Open→03Resolved a:03Dzahn [20:23:04] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Paladox) 05Resolved→03Open a:05Dzahn→03None [20:34:58] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Jdforrester-WMF) 05Open→03Resolved a:03Dzahn [20:37:38] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10serviceops, and 2 others: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Dzahn) @20after4 per recent talks we had.. is it still o... [20:41:57] 10Gerrit, 10Phabricator, 10Operations, 10Security-Team, 10Traffic: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Dzahn) [21:15:31] 10Release-Engineering-Team, 10serviceops: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Volans) 05Resolved→03Open I've noticed that Phabricator emails are failing the SPF check, re-opening to add details, feel free to move it to a separate task if needed.... [21:33:30] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) @Volans Thanks for reporting! The SPF record has been updated now in DNS. This should go away now. [23:50:02] 10Gerrit, 10Phabricator, 10ContentSecurityPolicy, 10Operations, and 2 others: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Bawolff) [23:54:28] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10good first task: Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10srishakatux) This doesn't look lik... [23:54:38] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912): Migrate fail-archived-repositories job to a new host - https://phabricator.wikimedia.org/T239979 (10srishakatux)