[01:21:17] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [03:06:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10ArielGlenn) @Ladsgroup what sort of issues have you had with yaml in python? How complicated are th... [03:57:34] it looks like the gate queue has gotten more pessimistic [03:57:41] it only runs the top 2 patches now? [04:02:02] so it seems [04:14:34] now it went 4 deep [04:18:58] weird [04:20:01] at 3 per batch (which it seems to be now) and 45 min per batch, see you in an hour and a half :-/ [04:41:04] they're all gone but one! [04:55:02] Yippee, build fixed! [04:55:03] Project mediawiki-core-code-coverage-docker build #4293: 09FIXED in 1 hr 55 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4293/ [06:16:43] 10Diffusion, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10hashar) [06:46:18] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:59:16] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10Core Platform Team Kanban (Doing), and 2 others: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 (10hashar... [07:01:52] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10MW-1.32-release, 10MW-1.33-release, and 2 others: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) This bug does affect MediaWiki 1.32 and is a regression.... [07:03:12] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10MediaWiki-Installer, 10MW-1.32-release, and 3 others: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) I forgot to loop #core_platform_team . This task is c... [07:03:20] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10MediaWiki-Installer, 10MW-1.32-release, and 3 others: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) p:05Triageβ†’03High [07:06:32] 10Continuous-Integration-Config, 10Code-Health-Metrics: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990 (10hashar) [07:10:34] 10Continuous-Integration-Config, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): mwext-MobileFrontend-npm-run-lint-modules-docker failing - node script running as php - https://phabricator.wikimedia.org/T196956 (10hashar) > Did anything come of this? It'd be nice to get rid of this hack, and if the ne... [07:17:06] 10Continuous-Integration-Config, 10Quibble: The phpunit-coverage jobs shouldn't run npm install - https://phabricator.wikimedia.org/T225008 (10hashar) That is an artifact from #Quibble, it has logic to skip installing dependencies (`--skip-deps`) but that also skip installing development dependencies which we... [07:20:19] (03PS1) 10Hashar: [wikiba.se] enable php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/514430 (https://phabricator.wikimedia.org/T224905) [07:21:32] (03PS3) 10Hashar: wikiba.se: drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [07:21:45] (03CR) 10Hashar: [C: 03+2] [wikiba.se] enable php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/514430 (https://phabricator.wikimedia.org/T224905) (owner: 10Hashar) [07:23:23] (03Merged) 10jenkins-bot: [wikiba.se] enable php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/514430 (https://phabricator.wikimedia.org/T224905) (owner: 10Hashar) [07:23:35] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10WMDE-leszek) Thanks gentlemen! I am a bit surprised to see PHPUnit 4.8 run happily on PHP 7.2 but as it works, why complain. I still... [07:30:48] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) >>! In T223971#5232629, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), hr... [07:32:06] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) p:05Triageβ†’03High Maybe it can be reproduced on the test machine `labtestvirt2003.codfw.wmnet` whic... [08:10:01] 10Phabricator, 10User-DannyS712: Herald: Rules request for DannyS712 - https://phabricator.wikimedia.org/T221574 (10DannyS712) @Aklapper any idea when this might by actioned? [08:11:05] 10Phabricator, 10User-DannyS712: Herald: Rules request for DannyS712 - https://phabricator.wikimedia.org/T221574 (10DannyS712) [08:16:19] (03PS9) 10Awight: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [08:16:38] (03CR) 10Awight: [C: 04-1] "PS9: Word on the street is that we actually do want to email the WMDE fundraising team. Patch adjusted accordingly." [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [08:17:08] (03CR) 10Awight: [C: 03+1] Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [08:23:17] (03CR) 10Hashar: [C: 03+2] backend: Fix likely typo in variable name [integration/quibble] - 10https://gerrit.wikimedia.org/r/513254 (owner: 10Legoktm) [08:24:02] (03Merged) 10jenkins-bot: backend: Fix likely typo in variable name [integration/quibble] - 10https://gerrit.wikimedia.org/r/513254 (owner: 10Legoktm) [08:24:34] (03CR) 10jenkins-bot: backend: Fix likely typo in variable name [integration/quibble] - 10https://gerrit.wikimedia.org/r/513254 (owner: 10Legoktm) [08:26:45] awight: I started looking at your Quibble changes, not sure when I will actually start +2ing some though :] [08:28:29] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >! In T224857#5233723, @thcipriani wrote: > > The speed of deployment directly relates to platform stability. Sacrif... [08:33:53] hashar: Right on, thanks! Yeah take your time, I tried to make each one safe. [08:51:59] 10MediaWiki-Codesniffer: Avoid use of isset( ..., ..., [...] ) function construct / syntax - https://phabricator.wikimedia.org/T219139 (10Tgr) Multi-parameter `unset()` seems harmless to me, and probably more readable than separate unset commands. Unlike `isset()` (where it is not obvious whether `isset( $x, $y... [08:59:31] PROBLEM - Host integration-slave-docker-1058 is DOWN: CRITICAL - Host Unreachable (172.16.6.2) [08:59:42] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) Ok... found another limitation. Pybal, after reading a change to etcd, waits one second before reading the next one (... [09:02:23] 10MediaWiki-Codesniffer: Avoid use of isset( ..., ..., [...] ) function construct / syntax - https://phabricator.wikimedia.org/T219139 (10D3r1ck01) @Tgr, thanks for your input. Looking at the documentation here: https://www.php.net/manual/en/function.isset.php, it's kinda clear that using multiple arguments tran... [09:04:34] RECOVERY - Host integration-slave-docker-1058 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms [09:07:10] !log Pooled in integration-slave-docker-1058 and integration-slave-docker-1059 [09:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:09:38] 10Project-Admins: Create a project for Wikilink - https://phabricator.wikimedia.org/T225065 (10Samwalton9) [09:22:43] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10zeljkofilipin) [09:25:31] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10hashar) [09:27:14] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10aborrero) p:05Triageβ†’03Normal [09:34:12] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10WMDE-Fisch) [09:35:14] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10WMDE-Fisch) I created a new task for the creation and impleme... [09:37:29] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10WMDE-Fisch) [09:41:38] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Ladsgroup) Deployment blocker: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/514446 this needs to backported to wm... [10:14:57] 10MediaWiki-Codesniffer: Avoid use of isset( ..., ..., [...] ) function construct / syntax - https://phabricator.wikimedia.org/T219139 (10Tgr) Having to look at the documentation to interpret something meets the definition of "not obvious". [10:44:53] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team (Code Health (TEC13)), and 3 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10Pginer-WMF) [11:16:22] 10Project-Admins: Create a project for Wikilink - https://phabricator.wikimedia.org/T225065 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #wikilink has been created: https://phabricator.wikimedia.org/project/view/4082/ Please encourage interested people to visit the project and to j... [11:26:02] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) >>! In T223102#5235257, @hdothiduc wrote: > You can actually remove me as a subscriber from... [11:33:52] 10Project-Admins: Create project for Wiki Techstorm 2019 - https://phabricator.wikimedia.org/T222929 (10Aklapper) 05Openβ†’03Resolved Thanks! * Performed the steps in T222929#5215579 * New public project #wikistorm-2019 has been created: https://phabricator.wikimedia.org/project/view/4083/ * **To do:** Sort... [12:50:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10hashar) I poked a different upstream meanwhile. @MoritzMuehlenhoff we would nee... [13:18:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:18:57] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) @Ladsgroup it's deployed, right? [13:20:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) group1 wikis at 1.34.0-wmf.8 [13:33:43] 10Scap, 10User-zeljkofilipin: Scap - https://phabricator.wikimedia.org/T225091 (10zeljkofilipin) [13:36:27] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Ladsgroup) >>! In T220733#5236718, @zeljkofilipin wrote: > @Ladsgroup it's deployed, right? > > https://tools.wmflabs.org/sal/log/AWsnS... [13:37:40] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10zeljkofilipin) p:05Triageβ†’03High [13:38:23] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10zeljkofilipin) [13:39:02] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Ladsgroup) [13:39:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Ladsgroup) p:05Triageβ†’03Unbreak! Definitely a train blocker. [13:39:40] 10Continuous-Integration-Config, 10Code-Health-Metrics: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990 (10kostajh) > Yes please hold. We lack capacity to run all those additional tests right now :-\ I am stuck not being able to add any more instances due to an ongoin... [13:42:30] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) Do we need to add `$wgMessageDirs` and `$wgExtensionMessagesFiles` to `FlaggedRevs.php` so they can be picked up when pulled via exten... [13:43:16] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10zeljkofilipin) p:05Highβ†’03Unbreak! @thcipriani do you know what went wrong? Is this blocking the train? If not, please decrease priority and remove from train blockers. [13:43:27] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) [13:43:30] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10zeljkofilipin) [13:43:53] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) >Very likely caused by making FlaggedRevs use extension-wip.json Is/Was it fine on .7? [13:44:14] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) Oh, ignore that. I couldn't see the revert in .7 of the wip patch [13:46:14] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10Reedy) ` reedy@mwdebug1001:/srv/mediawiki$ sudo -u mwdeploy rm -rf php-1.33.0-wmf.23/ reedy@mwdebug1001:/srv/mediawiki$ ` >>! In T225091#5236776, @zeljkofilipin wrote: > @thcipriani do you kn... [13:47:48] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10Reedy) ` reedy@mwdebug1001:/srv/mediawiki$ ls -al docroot/wwwportal/ total 16 drwxr-xr-x 4 mwdeploy mwdeploy 4096 Mar 19 2018 . drwxr-xr-x 10 mwdeploy mwdeploy 4096 Apr 15 15:43 .. lrwxrwxrwx... [13:47:54] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10zeljkofilipin) @Ladsgroup will this be fixed by train window [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190606T1300... [13:50:03] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10Reedy) >>! In T225091#5236803, @Reedy wrote: > ` > reedy@mwdebug1001:/srv/mediawiki$ ls -al docroot/wwwportal/ > total 16 > drwxr-xr-x 4 mwdeploy mwdeploy 4096 Mar 19 2018 . > drwxr-xr-x 10 mwd... [13:51:05] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10RazeSoldier) These tags are not suitable for subtask of train. [13:52:11] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Ladsgroup) >>! In T225092#5236804, @zeljkofilipin wrote: > @Ladsgroup will this be fixed by train window [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-201906... [13:52:27] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10JHedden) a:03JHedden [13:54:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy) [13:54:08] 10Scap, 10User-zeljkofilipin: `rsync error` during sync-pull-masters - https://phabricator.wikimedia.org/T225091 (10Reedy) 05Openβ†’03Resolved a:03Reedy ` reedy@mwdebug1001:/srv/mediawiki$ scap pull 13:53:40 Copying from deploy1001.eqiad.wmnet to mwdebug1001.eqiad.wmnet 13:53:40 Started rsync common 13:53:... [13:55:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) `lang=php $wgMessagesDirs['FlaggedRevs'] = __DIR__ . '/i18n/flaggedrevs'; $wgMessagesDirs['RevisionReview'] = __DIR__ . '/i18n/revisionreview'; $wgMessagesDirs['Stabiliza... [13:57:19] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Ladsgroup) >>! In T225092#5236838, @Reedy wrote: > `lang=php > $wgMessagesDirs['FlaggedRevs'] = __DIR__ . '/i18n/flaggedrevs'; > $wgMessagesDirs['RevisionR... [13:57:50] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10zeljkofilipin) [14:05:37] Amir1: ^ I wanted to favorite your comment there... [14:05:53] :D [14:06:57] heh [14:43:59] zeljkof: hey, I have a question regarding https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/514494 How can I disable the daily ruby browser tests? [14:44:22] Is just dropping them in extension wikibase enough? [14:44:59] this job? https://integration.wikimedia.org/ci/view/Selenium/job/selenium-Wikibase-chrome/ [14:46:20] Amir1: delete this project in integration/config https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/jjb/selenium.yaml#38 [14:46:22] 10Phabricator, 10User-DannyS712: Herald: Rules request for DannyS712 - https://phabricator.wikimedia.org/T221574 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Sorry, this task fell through the cracks. Thanks for the ping! I've created H322 for the first two items. There is [no need for a Herald rule for t... [14:46:29] and I can delete the job in jenkins [14:46:57] oh I was exactly looking for that. Thanks! [14:47:59] :) [14:48:01] no problemo [14:48:27] (03PS1) 10Ladsgroup: Drop daily ruby browser tests for wikibase and wikibase lexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514498 (https://phabricator.wikimedia.org/T224301) [14:51:33] zeljkof: https://gerrit.wikimedia.org/r/c/integration/config/+/514498 [14:52:46] (03CR) 10Zfilipin: [C: 03+2] Drop daily ruby browser tests for wikibase and wikibase lexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514498 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [14:54:28] (03CR) 10Zfilipin: [C: 03+2] "Deleted jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/514498 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [14:54:37] Amir1: +2d and deleted jobs in jenkins [14:54:50] \o/ [14:54:56] \o/ [14:55:02] Ruby is dead, long live nodejs [14:56:37] (03Merged) 10jenkins-bot: Drop daily ruby browser tests for wikibase and wikibase lexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514498 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [14:57:04] also, you're needed here (no rush though: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/512665) [15:06:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Jdforrester-WMF) a:03Reedy [15:08:57] Amir1: sorry, distracted with some urgent things now, train and a few more, will reply, but maybe not this week :( [15:12:37] no worries [15:13:51] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy) [15:13:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-FlaggedRevs, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) 05Openβ†’03Resolved Scap is still running, but FR is now in ExtensionMessages-1.34.0-wmf.8.php (but wasn't before), so should have... [15:19:09] (03PS1) 10Ladsgroup: Drop ruby browser tests from Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) [15:19:44] zeljkof: Another one ^ https://gerrit.wikimedia.org/r/#/c/integration/config/+/514507 [15:22:21] (03CR) 10Hashar: "I am not sure I am a fan of the additional Command layer and the build_params method when one can just invoke quibble.zuul.clone()." [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:29:11] (03PS2) 10Ladsgroup: Drop ruby browser tests from Wikibase and WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) [15:33:11] (03CR) 10Hashar: Extension and skin submodule update as command (034 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:35:29] (03CR) 10Awight: "> I am not sure I am a fan of the additional Command layer and the" [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:41:27] (03CR) 10Awight: Extension and skin submodule update as command (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:44:50] (03CR) 10Hashar: Composer dependencies as command (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513498 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:45:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-FlaggedRevs, 10User-zeljkofilipin: Missing i18n in FlaggedRevs - https://phabricator.wikimedia.org/T225092 (10Reedy) Confirmed fixed [15:48:28] (03PS2) 10Awight: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) [15:50:31] (03CR) 10Awight: Composer dependencies as command (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513498 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [15:51:14] (03PS2) 10Awight: Composer dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513498 (https://phabricator.wikimedia.org/T223752) [15:52:26] 10MediaWiki-Releasing, 10Security: Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205043 (10Reedy) 05Openβ†’03Resolved a:03Reedy [15:52:29] 10MediaWiki-Releasing, 10Security: Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205043 (10Reedy) [15:57:02] 10Release-Engineering-Team, 10Scap: Better error message than "Scap failed!: Call to mwscript eval.php returned: None" - https://phabricator.wikimedia.org/T225109 (10Reedy) [15:57:17] (03CR) 10Jforrester: [C: 03+2] Drop ruby browser tests from Wikibase and WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [15:59:15] (03Merged) 10jenkins-bot: Drop ruby browser tests from Wikibase and WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [16:01:24] !log Reloading Zuul to deploy 172a4e7886b285adb48a85dce7287dce5c3778e2 [16:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:09:03] okay, I have a good idea why CI got much slower recently [16:09:40] do tell? [16:10:02] we added a bunch of dependencies for phan to do static analysis against [16:10:12] and that significantly expanded the dependency map [16:10:22] and that list of deps is also used for phpunit etc [16:10:40] oooh [16:12:09] EventBus pulled in 64!!! dependencies including Wikibase, Scribunto, etc. just because it needs CentralNotice for phan purposes [16:12:30] o_0 [16:12:32] trees [16:16:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) p:05Triageβ†’03High [16:17:48] legoktm: That'd do it. [16:18:20] I've seen a few repos with custom phan stubs of the extensions they're working against, instead of dependencies. Can we do that more broadly? [16:19:43] that's one potential approach, but the downside with stubs is that you have to keep them updated, and then you don't have proper integration tests [16:19:51] True. [16:20:05] Clearly we need to move to a monorepo so everything is as slow as everything else. [16:23:40] you're not going to hear me object to a monorepo, I could get rid of so much custom tooling for batch operations [16:25:45] woo final release 1.27 announced, yay [16:25:47] if every CI build takes two hours and development pace slows to a crawl, we won’t break production as often either [16:25:59] haha [16:26:13] oh phan + phpunit reuse mmeeeeeehhhhh [16:26:50] so it's not just me, things really have been a lot slower [16:29:43] things are definitely slower [16:32:00] https://grafana.wikimedia.org/d/000000321/zuul?from=now-90d&to=now&panelId=14&fullscreen&orgId=1 [16:32:59] https://grafana.wikimedia.org/d/000000321/zuul?from=now-6M&to=now&panelId=14&fullscreen&orgId=1 it's slightly more pronounced in this view [16:33:11] the median used to be 16-17 minutes, now its around 25 min [16:34:48] I've just gotten unlucky and hit a lot of the 1 hour + ones [16:36:52] Lucas_WMDE: Or fix it as quickly. [16:37:42] James_F: nah, we’ll just bypass CI for any issues [16:37:48] I’ve thought this all through, you see [16:37:52] there are no downsides [16:37:53] apergos: Our one-day median is >1 hour. [16:37:54] ;) [16:38:21] oh man [16:38:24] that would be it then [16:38:30] Oh, or is that just me mis-reading? [16:38:37] Gate time vs. run time. [16:38:58] 1 hour waiting and 25 minutes of processing feels like 90 minutes to the human. [16:39:54] legoktm: Mono-repo means we essentially make the divide between Wikimedia and non-Wikimedia code unscalable. [16:40:08] legoktm: If anyone was going to object, I'd have thought it'd be you. ;-) [16:40:19] wouldn't it be like the old SVN days? [16:40:35] everything is still in folders, comparmentalized, but in one giant repo [16:40:41] No? We'd have a single repo called WikimediaProduction with core/skins/extensions. [16:40:42] so you can make changes to all extensions at once [16:40:49] We'd not add 2000 extensions to that repo, git would die. [16:41:03] So you'd either be inside the tent or left alone in the wilderness. [16:41:10] A pretty sucky world for third party code. [16:41:20] eh, plenty of giant monorepos exist [16:41:26] At Google and Facebook. [16:41:44] Who can afford a $300k build machine with 50TB of RAM. [16:41:47] ;-) [16:43:06] I haven't thought about the practicalities yet [16:43:17] And people at e.g. Wikia or WikiHow who want to hack on MediaWiki + twenty local extensions would have a very messy time. [16:43:26] I'd just really like to be able to make changes across all extensions at once instead of needing LU [16:43:30] Oh, sure. [16:43:38] I'd also like LU to be able to work again. ;-) [16:46:31] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): neither bureaucrats nor admins can grant the right for "autoreview" or "editor" since 1.34.0-wmf.8. - https://phabricator.wikimedia.org/T225115 (10Udo_T) [16:47:18] yeah, I was looking into how to manage a ssh key over the weekend [16:47:27] it's doableish [16:51:09] legoktm: I understand that your blocker will be removed "soon" anyway. [17:14:06] (03CR) 10Hashar: "This change does remove the job that runs rake test, which afaik is just for linting/rubocop." [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:22:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Umherirrender) It is possible to add a stub for the class from CentralNotice, so feel free to revert the config when the repository g... [17:23:17] legoktm: James_F: no idea why CentralNotice would pull in so much stuff... Maybe via the Translate extension? EventBus just registers a handler for a CN hook, and the unit tests cover parts that are independent of that IIRC [17:24:30] Project beta-scap-eqiad build #252165: 04FAILURE in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252165/ [17:24:31] AndyRussG: it's not CN, it's CN's recursive dependencies [17:24:47] CN -> MobileFrontend -> Wikibase [17:26:18] legoktm: ah gotcha [17:28:14] And MF touches pretty much everything. [17:28:15] CN hmmm I think the MobileFrontend dependency is not that important, dunno if any CN tests cover those bits. Also no idea what the criteria are for dependencies in CI (that's what's at issue here, right?) but pls lmk if there's stuff we can do to improve the situation [17:28:41] Well, we could certainly drop the CN->MF dependency, but that's not going to be that large a dent, is it? [17:28:51] (03PS1) 10Ladsgroup: Fix up for I8c55532bad85cff953436b5f5ed6137a61ea142c [integration/config] - 10https://gerrit.wikimedia.org/r/514547 (https://phabricator.wikimedia.org/T224301) [17:30:11] Amir1: Was about to ping you. What do you want to do? Restoring rake seems odd, but we should leave and make mwselenium-quibble-docker non-voting until it's fixed, yes? [17:30:16] (03CR) 10Ladsgroup: "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:30:42] James_F: I don't know TBH [17:30:46] I'm about to leave [17:30:51] late for something already [17:31:12] OK, I'll just fix things. [17:31:17] Have fun, speak tomorrow. [17:33:26] (03PS2) 10Jforrester: zuul: [Wikibase] Don't run mwselenium-quibble-docker for now [integration/config] - 10https://gerrit.wikimedia.org/r/514547 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:34:24] Project beta-scap-eqiad build #252166: 04STILL FAILING in 0.65 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252166/ [17:34:59] (03CR) 10Jforrester: "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:35:12] (03CR) 10Jforrester: [C: 03+2] zuul: [Wikibase] Don't run mwselenium-quibble-docker for now (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/514547 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:35:57] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547 [17:35:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:36:51] (03Merged) 10jenkins-bot: zuul: [Wikibase] Don't run mwselenium-quibble-docker for now [integration/config] - 10https://gerrit.wikimedia.org/r/514547 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [17:39:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) Yeah, @Jdforrester-WMF suggested that on IRC as one way to go. Do you have an opinion on potentially splitting the dependenc... [17:44:26] Project beta-scap-eqiad build #252167: 04STILL FAILING in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252167/ [17:46:27] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Jdforrester-WMF) Is this waiting for anything other than naming to be settled? [17:51:33] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Legoktm) Can we use the standard @small/@medium/@large tags (https://phpunit.de/manual/6.5/en... [17:54:23] Project beta-scap-eqiad build #252168: 04STILL FAILING in 0.69 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252168/ [17:59:40] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): neither bureaucrats nor admins can grant the right for "autoreview" or "editor" since 1.34.0-wmf.8. - https://phabricator.wikimedia.org/T225115 (10Udo_T) p:05Triageβ†’03High I changed the priority to "High"... [18:01:45] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Jdforrester-WMF) >>! In T225068#5237772, @Legoktm wrote: > Can we use the standard @small/@me... [18:04:27] Project beta-scap-eqiad build #252169: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252169/ [18:07:39] (03CR) 10Brennen Bearnes: [V: 03+1 C: 03+1] "I'm not a PhpStorm user, so I don't know that this works as such, but confirmed that it doesn't break anything obvious - seems fine to me." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513287 (owner: 10Kosta Harlan) [18:14:12] Krenair and/or thcipriani, can I safely move deployment-imagescaler03? [18:14:23] Project beta-scap-eqiad build #252170: 04STILL FAILING in 0.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252170/ [18:15:07] (03PS2) 10Brennen Bearnes: Add MediaWiki configuration for logging to stderr [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513195 (owner: 10Kosta Harlan) [18:17:26] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+1] "Tested - works well." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513195 (owner: 10Kosta Harlan) [18:20:15] Project beta-update-databases-eqiad build #34338: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34338/ [18:24:27] Project beta-scap-eqiad build #252171: 04STILL FAILING in 0.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252171/ [18:30:47] (03CR) 10Jeena Huneidi: [V: 03+1 C: 03+2] "Can't try the one but the Mac one works for me...well, I wrote it so if anyone else wants to test..." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514166 (https://phabricator.wikimedia.org/T223715) (owner: 10Brennen Bearnes) [18:32:50] andrewbogott: I don't know. Not sure how imagescalers are configured to work with mediawiki. Grepping puppet: similarly fruitless. [18:33:33] ok, I'll hold out for feedback from kren.air [18:34:25] Project beta-scap-eqiad build #252172: 04STILL FAILING in 0.74 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252172/ [18:36:08] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10JJMC89) [18:36:13] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): neither bureaucrats nor admins can grant the right for "autoreview" or "editor" since 1.34.0-wmf.8. - https://phabricator.wikimedia.org/T225115 (10JJMC89) [18:36:53] 10Release-Engineering-Team, 10MediaWiki-User-management: neither bureaucrats nor admins can grant the right for "autoreview" or "editor" since 1.34.0-wmf.8. - https://phabricator.wikimedia.org/T225115 (10JJMC89) p:05Highβ†’03Unbreak! Seems like a deployment blocker so raising to UBN [18:37:52] (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] "Thanks!" [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513195 (owner: 10Kosta Harlan) [18:41:24] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [18:41:40] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [18:43:08] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10hashar) >>! In T125050#5182872, @Anomie wrote: > ... > * Then there are the 1000+ tests in te... [18:43:35] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Ottomata) I don't think eventstreams is in k8s, is it? [18:44:26] Project beta-scap-eqiad build #252173: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252173/ [18:45:04] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) >>! In T198901#5237895, @Ottomata wrote: > I don't think eve... [18:45:43] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [18:46:01] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [18:46:36] (03CR) 10Hashar: "> mwselenium-legacy was already removed; I thought that mwselenium-quibble-docker was the new one? Still, it's not working yet, so removin" [integration/config] - 10https://gerrit.wikimedia.org/r/514507 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup) [18:48:53] (03CR) 10Hashar: "Skipping is being discussed on T225068" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll) [18:51:28] (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] "Haven't tested on phpStorm myself but if I recall correctly we worked on this during the hackathon - merging" [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513287 (owner: 10Kosta Harlan) [18:54:30] Project beta-scap-eqiad build #252174: 04STILL FAILING in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252174/ [19:04:24] Project beta-scap-eqiad build #252175: 04STILL FAILING in 0.64 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252175/ [19:10:48] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [19:14:36] Project beta-scap-eqiad build #252176: 04STILL FAILING in 0.63 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252176/ [19:16:51] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) [19:18:19] hmm [19:18:23] scap is dead [19:18:28] on beta [19:18:40] 00:00:00.515 RuntimeError: Scap failed!: Call to mwscript eval.php returned: None [19:18:41] bah [19:18:47] known problem of doom [19:19:28] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) p:05Triageβ†’03High seems also like a deployment blocker, therefore I rise the priority to "High". Please adjust if I... [19:20:16] Project beta-update-databases-eqiad build #34339: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34339/ [19:21:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10ayounsi) [19:21:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10ayounsi) 05Resolvedβ†’03Open Alert is warning again. > DISK WARNING - free space: / 4594 MB (10%... [19:22:26] $ /usr/bin/scap --verbose sync 'hashar test'usage: scap [-h] ... [19:22:26] scap: error: extra arguments found: --verbose [19:22:26] bah [19:23:28] hashar: I just emailed you about this too… can I move webperformance.integration.eqiad.wmflabs to a different host? Does it need to be depooled someplace? [19:24:24] Project beta-scap-eqiad build #252177: 04STILL FAILING in 0.58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252177/ [19:24:34] andrewbogott: about the strange cpu perf of some cloudvirt? [19:24:53] hashar: I just need to clear off cloudvirt1014 [19:25:17] andrewbogott: yeah I guess we can just move it [19:25:25] it is running a job right now, but that is a one hour + job [19:25:35] I can wait if it makes your life easier [19:25:36] and it just started, so might just kill it and eventually it will start again [19:25:45] there's nothing urgent happening [19:25:51] I am not sure what is the impact of killing it is. Most probably some gap in the performance data [19:25:57] then it might well just run continuously [19:26:15] 10Project-Admins: Create OpenRefine Phabricator Project - https://phabricator.wikimedia.org/T225142 (10Pintoch) [19:27:20] andrewbogott, thcipriani so I believe the swift upload frontend proxy talks to imagescalers [19:27:27] swift::proxy::thumborhost: deployment-imagescaler03.eqiad.wmflabs:8800 [19:27:29] is set on -ms-fe03 [19:30:27] hm, so maybe we can just point it to a different one? [19:30:28] 10Release-Engineering-Team, 10Scap: Better error message than "Scap failed!: Call to mwscript eval.php returned: None" - https://phabricator.wikimedia.org/T225109 (10hashar) Same as T224634 I clearly remember struggling with that error and I was blaming `scap.sh`. T222372 is about removing that code entirely.... [19:30:32] * andrewbogott checks to see if there are different ones [19:31:03] there are [19:32:57] I think if the instance goes down, uncached thumbnailing of images on upload.beta.wmflabs.org will stop working [19:33:22] which is probably not that bad [19:33:25] it's only a small instance [19:33:28] is that something that anyone will notice/care about? It wont' be for very long [19:33:36] yeah, ok, I'll just be bold [19:34:04] Reedy: James_F: FlaggedRevs is broken on beta cluster due to extension registry conversion I believe or maybe that is operations/mediawiki-config not being up to date yet. Filled as an UBN https://phabricator.wikimedia.org/T225143 [19:34:15] oh sorry that one is medium [19:34:16] still [19:34:18] Reedy: James_F: Fatal error '/srv/mediawiki-staging/php-master/extensions/Flagged Revisions/extension.json does not exist!' [19:34:26] lol [19:34:29] Project beta-scap-eqiad build #252178: 04STILL FAILING in 0.73 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252178/ [19:34:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Udo_T) [19:34:52] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) [19:35:17] * @param string $ext Name of the extension to load [19:35:33] legoktm: Is that the dirname, rather than the "name" in extension.json? [19:35:35] I guess so... [19:35:38] ;] [19:35:53] It's confusing, as it's Flagged Revisons in extension.json [19:36:21] Why am I asking when the code is there [19:36:21] :P [19:36:22] $path = "$wgExtensionDirectory/$ext/extension.json"; [19:36:57] patch incoming [19:37:31] heh, 2 minutes to copy [19:38:11] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Cirdan) [19:38:26] Reedy: yeah its dirname. Both should really be the same, there are some assumptions in code that isn't used yet that they are [19:38:46] I think I should probably fix the "name" in FR to FlaggedRevs then rather than Flagged Revisions [19:40:58] andrewbogott, 2 minutes to move the instance? [19:41:09] I guess the imagescalers themselves don't store much. . . [19:41:10] yeah, it's done already [19:41:16] seems not! [19:43:04] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Cirdan) p:05Highβ†’03Unbreak! This is very likely related to T225115, there is something wrong with the FlaggedRevs configur... [19:43:43] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Cirdan) [19:43:59] These really aren't UBN [19:44:24] Project beta-scap-eqiad build #252179: 04STILL FAILING in 0.59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252179/ [19:45:13] https://gerrit.wikimedia.org/r/514561 if someone wants to unbreak beta [19:46:00] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy) [19:47:18] Reedy: thank you :] [19:47:30] Reedy: let's see how long it takes to wind through the jenkins jobs :) [19:47:36] lol [19:47:37] +2,+1 applied [19:47:44] Ta [19:47:49] As for all the FR revs [19:48:00] s/revs/config issues/ [19:48:03] This just shows how janky and incosistent it is [19:48:11] yeah I am fine [19:48:16] blame aaron ;) [19:48:17] they are hard to pull right [19:48:36] haven't we reverted the conversion to extension registry that was done during the hackathon? [19:48:44] iirc it was severly broken [19:49:06] No, we fixed most of the issues [19:49:22] I prefer to just play whack-a-mole to get it fixed now [19:49:28] It's a PITA to test [19:49:36] So it's easier to break the wikis and then fix up the config to match [19:49:52] If we keep just reverting it (when it's not completely broken), it just delays it [19:50:00] agree [19:50:16] 10Diffusion, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10mmodell) Worked on this today with Tyler in our pairing session. @halfak: This s... [19:50:17] I think the first revert was definitely sensible, as it had numerous big issues that needing sorted [19:50:19] *sorting [19:50:53] then next week we do not [19:50:58] then next week we do not do any deployment [19:51:13] so at least that buys us time for someone to test FlaggedRevs on beta and polish it up [19:51:26] though the config settings are probably different / not used on beta :-\ [19:52:23] Reedy: anyway in short: thank you for actively migrating extensions to extension registry and twice thank you for being super reactive dealing with the aftermath of such conversions! [19:52:45] bd808: thank you for the FlaggedRevs review/+2 :] [19:53:42] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Reedy) I'm sorry for the inconvenience, it's partly because the FR config is beyond a mess @Udo_T Which one do you expect (pr... [19:53:49] I'm generally happy if I break stuff to try and fix them ;P [19:54:27] Project beta-scap-eqiad build #252180: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252180/ [19:55:01] bd808 / greg-g could one of you please add stashbot to #wikimedia-codehealth ? [19:58:08] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Reedy) [19:59:16] kostajh: sure :) [20:00:00] I wonder if the FR config should just end up in some closure... [20:00:09] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) Hi @Reedy, in the above picture we formerly didn't have the line with "Status gesichtet (patrouilliert) Tiefe: einf... [20:02:38] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) I think $wgFlaggedRevsTags['accuracy']['levels'] = 1; is the correct config for us [20:03:43] bd808: working great, thank you [20:04:26] 10Phabricator: Move various acl* projects as subprojects to Policy-Admins - https://phabricator.wikimedia.org/T224818 (10mmodell) 05Invalidβ†’03Declined [20:04:45] Project beta-scap-eqiad build #252181: 04STILL FAILING in 0.64 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252181/ [20:05:19] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Reedy) `lang=php reedy@deploy1001:/srv/mediawiki-staging$ mwscript eval.php dewiktionary > var_dump( $wgFlaggedRevsTags ); a... [20:07:08] 10Diffusion, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10mmodell) [20:08:39] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) Ok, I take the picture in English from the other task: we formerliy didn't had the line with Accuracy, Depth and Readab... [20:12:28] Bleugh [20:12:40] So the problem we have is when FR comes in and overrides the default config [20:12:47] well, extension.json does [20:12:48] And brings that in [20:13:43] oh that's nice [20:14:28] Project beta-scap-eqiad build #252182: 04STILL FAILING in 3.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252182/ [20:15:05] https://phabricator.wikimedia.org/T142663 dejavu? ;) [20:15:37] > Defaults should be sane [20:15:38] lol [20:15:48] ohsigh [20:18:08] Sooooo [20:18:18] Do I just change the default in flagged revs? [20:18:54] If none of the WMF wikis are using it [20:18:57] I'd suggest that means it's not sane [20:20:13] Project beta-update-databases-eqiad build #34340: 04STILL FAILING in 13 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34340/ [20:20:18] Or deffering the config [20:20:54] legoktm: Any bright ideas/preferred solutions? ;P [20:22:57] Reedy: the quick hack for now would be to unset in a $wgExtensionFunctions, the real fix would be to make the config structure a bit more sane so we can do $wgWhatver['isntwanted'] = false [20:24:24] Project beta-scap-eqiad build #252183: 04STILL FAILING in 0.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252183/ [20:25:26] there's the fix for Right Now and then there's two weeks of no deploy (isn't it two weeks? next week and then isn't there something else after that?) for the better fix [20:26:08] I'll just unset in extension functions [20:26:14] seems the easiest way forward [20:36:24] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Daimona) I'm not a fan of stubs either. I think splitting PHPUnit and phan is the right way to do this. Temporarily using stubs is al... [20:36:52] Yippee, build fixed! [20:36:52] Project beta-scap-eqiad build #252184: 09FIXED in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/252184/ [20:40:54] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Reedy) How does it look now? `lang=php reedy@deploy1001:/srv/mediawiki-staging$ mwscript eval.php dewiktionary > var_dump( $w... [20:41:18] it's like whack-a-right [20:42:38] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Udo_T) Hi @Reedy, now it looks the same as before. That's what we wanted. Udo T. [20:42:55] now nobody touch it.... [20:42:58] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy) [20:43:01] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs multiple levels of review status since 1.34.0-wmf.8 (81a18d9) - https://phabricator.wikimedia.org/T225138 (10Reedy) 05Openβ†’03Resolved a:03Reedy [20:43:21] Most of that copied in config at the top probably can come out again eventually [20:45:32] resisting the urge to go look at it and figure out how things ought to be [20:45:34] nopenopenope [20:46:52] Reedy: beta cluster fixed :] THANK YOU! [20:48:05] apergos: I'd refrain from looking at that for a few weeks [20:48:19] like after .9 is everywhere, and we can update wmf-config a bit [20:48:35] I"m not going to look. I already have a plate of ocd urgings, heaped full [21:02:06] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:03:11] 10Phabricator (Upstream), 10Upstream: Access-restricted Phab applications appear in quick search results for all users, give "Access Denied" - https://phabricator.wikimedia.org/T94268 (10mmodell) 05Openβ†’03Resolved a:03epriestley Looks fixed to me, thanks @epriestley [21:07:50] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:12:08] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:14:16] 10Phabricator (Upstream), 10Upstream: Shortcut keys for description fields text formatting are needed. - https://phabricator.wikimedia.org/T816 (10mmodell) Another possibility would be something like https://addons.mozilla.org/en-US/firefox/addon/shortkeys/ [21:21:17] Yippee, build fixed! [21:21:17] Project beta-update-databases-eqiad build #34341: 09FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34341/ [21:24:54] 10MediaWiki-Releasing, 10Security: Test - https://phabricator.wikimedia.org/T225155 (10Reedy) [21:25:07] 10MediaWiki-Releasing, 10Security: Test - https://phabricator.wikimedia.org/T225155 (10Reedy) [21:25:13] 10MediaWiki-Releasing, 10Security: Test - https://phabricator.wikimedia.org/T225155 (10Reedy) 05Openβ†’03Invalid [21:33:30] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:54:59] 10Continuous-Integration-Config, 10Browser-Tests: Tracking: flapping tests in gated extensions - https://phabricator.wikimedia.org/T225162 (10awight) [21:57:57] 10Continuous-Integration-Config, 10Browser-Tests: Spurious timeout in core browser test - https://phabricator.wikimedia.org/T225163 (10awight) [22:09:52] 10Continuous-Integration-Config, 10Browser-Tests: Tracking: flapping tests in gated extensions - https://phabricator.wikimedia.org/T225162 (10awight) [22:16:34] Project beta-code-update-eqiad build #249476: 04FAILURE in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/249476/ [22:17:01] 23:16:33 fatal: unable to access 'https://gerrit.wikimedia.org/r/wikimedia/portals.git/': The requested URL returned error: 503 [22:18:06] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Phabricator email comments not posted - https://phabricator.wikimedia.org/T224752 (10mmodell) {rOPUPc23fe1fc58ac96f649625dda358b1b84abdad022} [22:18:30] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10Browser-Tests: Potentially fragile test in GrowthExperiments - https://phabricator.wikimedia.org/T225165 (10awight) [22:20:57] 10Continuous-Integration-Config, 10Browser-Tests: Tracking: flapping tests in gated extensions - https://phabricator.wikimedia.org/T225162 (10awight) [22:21:37] 10Continuous-Integration-Config, 10Browser-Tests: Spurious timeout in MediaWiki's rollback browser test - https://phabricator.wikimedia.org/T225163 (10Jdforrester-WMF) [22:22:08] Reedy: Given that all of gerrit is down, that's unsurprising. [22:22:29] 10Gerrit: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10crusnov) [22:24:24] Yippee, build fixed! [22:24:25] Project beta-code-update-eqiad build #249477: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/249477/ [22:24:40] 10Continuous-Integration-Config: Stall and timeout during PHPUnit tests - https://phabricator.wikimedia.org/T225167 (10awight) [22:25:47] 10Continuous-Integration-Config: Stall and timeout during PHPUnit tests - https://phabricator.wikimedia.org/T225167 (10Jdforrester-WMF) No, that's the symptom; it just means it was still doing a PHPUnit thing when it hit the overall hard-limit of 60 mins. It's almost always caused by composer (packagist) or node... [22:26:24] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10JHedden) Some initial baseline data: | **host** | **type** | **time test avg** | **stress-ng** | **sysbench** |... [22:32:13] 10Gerrit: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10CDanis) Some curious stuff in the monitoring data: * First Icinga notification at 22:10:26 UTC for a socket timeout after 10 seconds for "Gerrit JSON" * 22:10 was also the time of the first user complaint * Host-level sta... [22:33:59] 10Continuous-Integration-Config: Stall and timeout during PHPUnit tests - https://phabricator.wikimedia.org/T225167 (10awight) >>! In T225167#5238653, @Jdforrester-WMF wrote: > No, that's the symptom; it just means it was still doing a PHPUnit thing when it hit the overall hard-limit of 60 mins. It's almost alwa... [22:34:53] 10Continuous-Integration-Config: Stall and timeout during PHPUnit tests - https://phabricator.wikimedia.org/T225167 (10Jdforrester-WMF) Oh, system time vs. wall time on a contended resource? Eh. [23:36:16] Anyone knows if there's a way to create a dashboard on the new gerrit? [23:36:32] switch back to the old ui, create it, and then switch back? ;P [23:36:44] Even that doesn't work :\ all my previous dashboards are broken [23:37:10] I asked a while back and was told dashboards will be available, I am not sure if there's another version to wait up on or if it's available and I just don't know how to create them? [23:37:31] https://gerrit.wikimedia.org/r/Documentation/user-dashboards.html [23:37:56] They should be there, in some form at lesat [23:38:22] oooh [23:38:31] OK, so I'll need to build this query [23:41:48] 10Gerrit: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10thcipriani) I have gc info from right when this happened: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDYvNS8tLWp2bV9nYy5nZXJyaXQubG9nLjcuY3VycmVudC0tMjMtMjgtNTE=&channel=WEB Looks like there was a sudden spike i... [23:44:22] Hm, it keeps redirecting my to my own dashboard. [23:45:59] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Browser-Tests: Spurious timeout in MediaWiki's rollback browser test - https://phabricator.wikimedia.org/T225163 (10Reedy) [23:46:04] link? [23:49:11] hmm it looks like we can set up a project/team dashboard with a definition file, too [23:49:27] oh, uh here's what I tried: https://gerrit.wikimedia.org/#/dashboard/?title=Anti+Harassmet+Team&Work+by+team=owner:thalia.e.chan%2540googlemail.com+OR+owner:moriel%2540gmail.com+OR+owner:dbarratt%2540wikimedia.org+OR+owner:dmaza%2540wikimedia.org+AND+status:open [23:49:33] Should be the team's open patches [23:49:48] that url works fine for me [23:50:02] It takes you to a dashboard?! [23:50:06] Yup [23:50:09] They're not all open... [23:50:15] https://gerrit.wikimedia.org/r/dashboard/self <-- it takes me here [23:50:16] But it's definitely not my usual dashboard [23:50:29] I see commits going back to 2013 :) [23:50:33] It redirects me [23:51:04] works in incognito too [23:51:30] $ curl -I "https://gerrit.wikimedia.org/r/#/dashboard/?title=Anti+Harassmet+Team&Work+by+team=owner:thalia.e.chan%2540googlemail.com+OR+owner:moriel%2540gmail.com+OR+owner:dbarratt%2540wikimedia.org+OR+owner:dmaza%2540wikimedia.org+AND+status:open" [23:51:30] HTTP/1.1 200 OK [23:51:44] ... [23:51:47] the hell [23:51:58] mooeypoo: you apparently can't spell Harassment though ;) [23:52:14] haha [23:52:22] ... maybe that's why it's punishing m [23:52:24] me* [23:53:09] ...wait [23:53:21] you said incognito window -- are you seeing this in the OLD ui or the new one? [23:53:37] I can see it in the old UI, but it's not working in the new one -- it redirects me to my dashboard instead [23:53:39] old [23:53:45] the problem is with the new one :\ [23:53:52] paladox: Is there something broken with the new ui and custom dashboards? [23:54:07] mooeypoo dashboard support will be in 2.16 (for the new ui) [23:54:13] Reedy yup [23:54:23] paladox: ah! I wasn't sure if it's in yet or not [23:54:24] gerrit 2.15 dosn't support project dashboards [23:54:24] mooeypoo: I guess that's what you were thinking of then [23:54:39] https://gerrit.wikimedia.org/r/Documentation/user-dashboards.html#project-default-dashboard suggests otherwise though [23:54:44] and fun fact, the dashboard support in polygerrit (from 2.16) is the cause for breaking /p/ :P [23:54:47] paladox: would it be something we individually can make, or would it be like the old UI where we need to set up a config for the project to have a "project dashboard" [23:55:03] mooeypoo i have an issue to support this :) [23:55:13] you mean to fix this? [23:55:18] or an issue that blocks this? [23:55:35] mooeypoo dashboards are supported in a newer version for (polygerrit) :) [23:55:57] Any idea when we'll get that newer version in for wikimedia gerrit? [23:56:26] i think the plan is *soon*, but no fixed time. [23:56:43] meh. I might have to go back to the old ui just for the dashboard support [23:56:49] though the priority on it was bumped due to finding something in gerrit that can only be configured in the newer version. [23:56:53] paladox: thanks for all the work you're doing on this! [23:57:01] your welcome :) [23:57:03] oh, that's encouraging [23:57:28] Alright, I'll try to be patient, then. [23:57:31] ;) thanks! [23:58:08] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments: Potentially fragile test in GrowthExperiments - https://phabricator.wikimedia.org/T225165 (10SBisson) (Not a browser test) We've seen it failing for the first time today and then it passed on recheck. There's indeed something fragile about... [23:58:39] mooeypoo https://phabzilla.wmflabs.org/file/data/ttod654h24cvkwny2rhc/PHID-FILE-vbusoymcis3p7e7p5ur6/Screenshot_2019-03-12_at_23.27.55.png [23:58:46] that's the dashboard support in gerrit :) [23:58:58] (i didn't configure the dashboard though, so its empty)