[00:02:11] (03CR) 10Ahmon Dancy: [C: 04-1] sync-world: Change handling of wikiversions.php (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/757759 (owner: 10Ahmon Dancy) [00:05:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: publishing to doc.wikimedia.org fails: rsync command not found - https://phabricator.wikimedia.org/T300236 (10bd808) [00:05:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10bd808) [00:05:37] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) End-of-workday notes before I head out for an appointment: Currently seems stable on all wikis. Logs are current... [03:53:12] Yippee, build fixed! [03:53:13] Project mediawiki-core-phpmetrics-docker build #1161: 09FIXED in 4 min 11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/1161/ [04:48:15] (03CR) 10Krinkle: Adjust cache settings for more performant browser tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/738252 (owner: 10Kosta Harlan) [05:44:28] (03PS1) 10Gergő Tisza: Add Iniquity to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/757795 [05:53:22] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10IN) [05:54:15] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10IN) [05:55:37] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10IN) [05:58:14] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10Peachey88) @IN please don't not make random edits to tasks, These are disruptive to multiple users and alters the history of tasks, You can test and play at https://phab.wmflabs.org/ and learn about phabricator... [05:58:21] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10JJMC89) 05duplicate→03Invalid [07:07:26] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10IN) >>! In T300328#7658562, @Peachey88 wrote: > @IN please don't not make random edits to tasks, These are disruptive to multiple users and alters the history of tasks, You can test and play at https://phab.wmf... [08:00:51] Yippee, build fixed! [08:00:52] Project Wikibase-phpmetrics-docker build #480: 09FIXED in 49 sec: https://integration.wikimedia.org/ci/job/Wikibase-phpmetrics-docker/480/ [08:16:49] good morning [08:18:20] (03CR) 10Kosta Harlan: Run post-dependency install, pre-test steps in parallel (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (https://phabricator.wikimedia.org/T225730) (owner: 10Kosta Harlan) [08:23:00] hashar: hi, are you OK with deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/757792 today? needed to fix T300336 [08:23:01] T300336: Add an image: Mobile-only quality gate shown to desktop users - https://phabricator.wikimedia.org/T300336 [08:24:57] kostajh: yessss [08:25:15] I am always fine deploying stuff when the developer is around, wanna pair on it now? [08:25:46] hashar: yes, just give me a minute to verify that it actually fixes the issue [08:26:08] we can also live test it on prod via mwdebug [08:26:57] I cant really tell though what is the impact of adding a new default user option [08:27:12] I guess any users save end up adding an extra propriety to the database [08:27:23] then it is probably not a problem :D [08:36:24] (03CR) 10Hashar: [C: 03+2] "I thought about having phpbench to a standalone image much like our image releng/quibble-fresnel image. It takes care of npm installing fr" [integration/quibble] - 10https://gerrit.wikimedia.org/r/741974 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:36:42] half an hour to review a change :D [08:41:08] hashar: can you quickly review https://gerrit.wikimedia.org/r/c/operations/puppet/+/756989 please? [08:46:55] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) 05Open→03Resolved scap was updated to 4.2.1 everywhere [08:49:08] kostajh: I am in a call, let me know when you wanna deploy the patch [08:49:21] (03CR) 10Awight: Run post-dependency install, pre-test steps in parallel (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (https://phabricator.wikimedia.org/T225730) (owner: 10Kosta Harlan) [08:50:45] (03Merged) 10jenkins-bot: phpbench: Support aggregate reports [integration/quibble] - 10https://gerrit.wikimedia.org/r/741974 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:55:44] taavi: i will check, but usually verified label is not for human [09:05:51] hashar: ok I thik we can go ahead with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/757792 when you're ready. [09:06:41] (03CR) 10Hashar: "The puppet compiler can be triggered by commenting "check experimental" in Gerrit which cause Zuul to trigger the compiler and it reports " [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 (owner: 10Majavah) [09:06:59] taavi: it is not straight forward unfortunately [09:07:37] taavi: it would probably be better to do a first pass of discussion via task and exchange about the envisaged workflow, then once we know what we wanna do / implement, do the related config change :D [09:07:41] verified label is a bit touchy [09:07:46] kostajh: sure [09:08:00] kostajh: lets do that in -operations [09:08:02] ack, thanks [09:08:14] will think about it a bit and come back to you [09:12:13] taavi: please cc me on the discussion :) [09:12:18] I am more than happy to help [09:12:45] thank you! [09:30:44] 10Quibble: Use Memcached with Quibble - https://phabricator.wikimedia.org/T300340 (10kostajh) [09:31:30] (03PS1) 10Kosta Harlan: Set Memcached as main cache type if extension is loaded [integration/quibble] - 10https://gerrit.wikimedia.org/r/757844 (https://phabricator.wikimedia.org/T300340) [09:37:17] (03PS1) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) [09:45:19] (03CR) 10Kosta Harlan: [C: 04-2] Adjust cache settings for more performant browser tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/738252 (owner: 10Kosta Harlan) [09:54:37] (03PS2) 10Kosta Harlan: Set Memcached as main cache type if extension is loaded [integration/quibble] - 10https://gerrit.wikimedia.org/r/757844 (https://phabricator.wikimedia.org/T300340) [10:00:09] kostajh: the phpbench aggregate patch has merged! [10:04:21] (03CR) 10Jbond: [C: 03+1] Review access change (031 comment) [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 (owner: 10Majavah) [10:18:37] (03CR) 10Hashar: [V: 03+2 C: 03+2] "Spoke with John about it, I got a bit confused earlier this morning but we have clarified the concerns. Note the Verified vote is restri" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 (owner: 10Majavah) [10:19:33] taavi: clarified concerns with jbond over a chat :] [10:19:35] it is in [10:36:56] (03Abandoned) 10Kosta Harlan: GitClean: Default to background processing [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 (owner: 10Kosta Harlan) [11:12:40] taavi: pcc users can now vote Verified+1 :] thank you for the gerrit change! [11:26:34] 10Phabricator: The task cannot be closed as a duplicate - https://phabricator.wikimedia.org/T300328 (10Aklapper) @IN: Please do not vandalize here, or get your account disabled. Thanks. [12:00:30] (03PS2) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) [12:02:18] (03PS1) 10Kosta Harlan: Remove Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/757888 [12:29:57] (03PS3) 10Kosta Harlan: Set Memcached as main cache type if extension is loaded [integration/quibble] - 10https://gerrit.wikimedia.org/r/757844 (https://phabricator.wikimedia.org/T300340) [12:30:36] (03PS4) 10Kosta Harlan: Set Memcached as main cache type if extension is loaded [integration/quibble] - 10https://gerrit.wikimedia.org/r/757844 (https://phabricator.wikimedia.org/T300340) [13:12:15] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10hnowlan) I think I'm still seeing this in 4.2.1 I'm afraid: ` hnowlan@restbase1020:~$ sudo -u deploy-service /usr/bin/scap deploy-local --repo cassandra/twcs -D log_json:F... [13:19:25] (03CR) 10Hashar: [C: 03+1] "It is more or less abandon ware indeed." [integration/quibble] - 10https://gerrit.wikimedia.org/r/757888 (owner: 10Kosta Harlan) [13:45:05] (03CR) 10Awight: "Fine with me, I wrote this only to prototype the apache changes..." [integration/quibble] - 10https://gerrit.wikimedia.org/r/757888 (owner: 10Kosta Harlan) [13:56:19] awight: I guess you can +2 https://gerrit.wikimedia.org/r/c/integration/quibble/+/757888/ :D [13:56:23] GRR [13:56:34] stupid bot, I will look at having that wikibugs patch deployed [14:02:13] taavi: hi again, looks like you are a member of `tools.wikibugs` would you have the capacity to approve and deploy a config change for it please? https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/757403 [14:02:26] that is to move stuff related to Quibble outside of this -releng channel :] [14:02:30] deploy guide https://www.mediawiki.org/wiki/Wikibugs#Deploying_changes [14:02:39] lemme see [14:03:16] I was added just so I can restart it when it broke (this was way before I had those powers as a Toolforge admin), I don't know at all how it works but this seems rather straightforward [14:05:45] taavi: oh don't worry , I can always ping others :] [14:08:14] hashar: done I think [14:08:54] hurrah thank you [14:09:09] can you test it works properly? [14:09:23] (03CR) 10Hashar: [C: 03+2] Add Iniquity to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/757795 (owner: 10Gergő Tisza) [14:11:05] (03CR) 10Hashar: [C: 03+2] Add .editorconfig [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756025 (owner: 10Ahmon Dancy) [14:11:13] (03Merged) 10jenkins-bot: Add Iniquity to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/757795 (owner: 10Gergő Tisza) [14:11:55] (03Merged) 10jenkins-bot: Add .editorconfig [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756025 (owner: 10Ahmon Dancy) [14:19:19] (03CR) 10Hashar: "I apologize, I am lagging in my review. Given the repository uses Python I guess we should instead add `commit-message-validator` as a tes" [integration/config] - 10https://gerrit.wikimedia.org/r/756042 (owner: 10BryanDavis) [14:23:44] (03CR) 10Hashar: [C: 03+2] "I apologize I am really lagging in my reviews" [integration/config] - 10https://gerrit.wikimedia.org/r/755734 (owner: 10Robert Vogel) [14:25:51] (03Merged) 10jenkins-bot: Add `BlueSpiceProDistributionConnector` [integration/config] - 10https://gerrit.wikimedia.org/r/755734 (owner: 10Robert Vogel) [14:26:17] there are some stuck jobs in Jenkins I will take a heap dump [14:27:48] !log taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx` [14:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:28:38] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 (1... [15:18:17] (03CR) 10Jforrester: [C: 04-1] "You can't bump the real version numbers of these images – they're mirroring the version of quibble they include. Instead they should be 1." [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [15:24:45] (03CR) 10BryanDavis: zuul: [wikimedia/toolhub] Add commit-message-validator to tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/756042 (owner: 10BryanDavis) [15:27:59] Greetings [15:37:21] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10dancy) Sorry about this @hnowlan I will investigate more deeply. [15:40:08] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10dancy) 05Resolved→03In progress [15:40:10] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [16:24:59] (03PS3) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) [16:25:21] (03CR) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [16:34:43] 10Release-Engineering-Team, 10Scap: scap should automatically update mediawiki.org roadmap pages - https://phabricator.wikimedia.org/T300384 (10Majavah) [16:54:25] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:00:10] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:20:53] (03PS1) 10Ahmon Dancy: scap deploy: Fix -D option handling (really this time) [tools/scap] - 10https://gerrit.wikimedia.org/r/757933 (https://phabricator.wikimedia.org/T300177) [17:21:37] (03CR) 10jerkins-bot: [V: 04-1] scap deploy: Fix -D option handling (really this time) [tools/scap] - 10https://gerrit.wikimedia.org/r/757933 (https://phabricator.wikimedia.org/T300177) (owner: 10Ahmon Dancy) [17:23:31] (03PS2) 10Ahmon Dancy: scap deploy: Fix -D option handling (really this time) [tools/scap] - 10https://gerrit.wikimedia.org/r/757933 (https://phabricator.wikimedia.org/T300177) [17:24:47] (03CR) 10Ahmon Dancy: [C: 03+2] scap deploy: Fix -D option handling (really this time) [tools/scap] - 10https://gerrit.wikimedia.org/r/757933 (https://phabricator.wikimedia.org/T300177) (owner: 10Ahmon Dancy) [17:26:01] (03Merged) 10jenkins-bot: scap deploy: Fix -D option handling (really this time) [tools/scap] - 10https://gerrit.wikimedia.org/r/757933 (https://phabricator.wikimedia.org/T300177) (owner: 10Ahmon Dancy) [17:27:38] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) Thanks @JMeybohm ! [17:50:01] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:58:20] (03PS1) 10Ahmon Dancy: scap deploy: misc tweaks [tools/scap] - 10https://gerrit.wikimedia.org/r/757942 [18:08:20] (03PS1) 10SBassett: Add more tags and attributes to i18n security checker allow list [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) [18:09:14] (03CR) 10Dduvall: [C: 04-1] "Sorry for the delay in reviewing this and thanks for the submission!" [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [18:09:25] (03CR) 10jerkins-bot: [V: 04-1] Add more tags and attributes to i18n security checker allow list [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [18:13:33] (03PS2) 10SBassett: Add more tags and attributes to i18n security checker allow list [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) [18:19:16] (03PS1) 10Ahmon Dancy: Release 4.2.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757947 [18:20:03] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.2.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757947 (owner: 10Ahmon Dancy) [18:21:17] (03Merged) 10jenkins-bot: Release 4.2.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757947 (owner: 10Ahmon Dancy) [18:22:41] (03PS1) 10Ahmon Dancy: Updated release procedure [tools/scap] - 10https://gerrit.wikimedia.org/r/757948 [18:23:43] (03PS2) 10Ahmon Dancy: Updated release procedure [tools/scap] - 10https://gerrit.wikimedia.org/r/757948 [18:23:59] (03CR) 10Ahmon Dancy: [C: 03+2] Updated release procedure [tools/scap] - 10https://gerrit.wikimedia.org/r/757948 (owner: 10Ahmon Dancy) [18:25:04] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:26:03] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10dancy) [18:26:33] (03Merged) 10jenkins-bot: Updated release procedure [tools/scap] - 10https://gerrit.wikimedia.org/r/757948 (owner: 10Ahmon Dancy) [18:26:35] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10dancy) [18:26:39] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10dancy) [18:43:27] (03PS4) 10Ahmon Dancy: sync-world: Change handling of wikiversions.php [tools/scap] - 10https://gerrit.wikimedia.org/r/757759 [18:44:09] (03CR) 10Ahmon Dancy: [C: 03+1] sync-world: Change handling of wikiversions.php [tools/scap] - 10https://gerrit.wikimedia.org/r/757759 (owner: 10Ahmon Dancy) [20:46:32] (03CR) 10Thcipriani: [C: 03+2] "Good call on this." [tools/scap] - 10https://gerrit.wikimedia.org/r/757942 (owner: 10Ahmon Dancy) [20:47:12] (03Merged) 10jenkins-bot: scap deploy: misc tweaks [tools/scap] - 10https://gerrit.wikimedia.org/r/757942 (owner: 10Ahmon Dancy) [20:48:08] (03CR) 10Ahmon Dancy: scap deploy: misc tweaks (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/757942 (owner: 10Ahmon Dancy) [20:57:01] (03PS1) 10Ahmon Dancy: scap deploy: misc tweaks followup [tools/scap] - 10https://gerrit.wikimedia.org/r/757978 [20:58:41] (03CR) 10Ahmon Dancy: [C: 03+2] scap deploy: misc tweaks followup [tools/scap] - 10https://gerrit.wikimedia.org/r/757978 (owner: 10Ahmon Dancy) [20:59:22] (03Merged) 10jenkins-bot: scap deploy: misc tweaks followup [tools/scap] - 10https://gerrit.wikimedia.org/r/757978 (owner: 10Ahmon Dancy) [21:35:17] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) Still looking stable. Optimistically calling this resolved. Thanks, as ever, to the many folks who helped out... [21:35:19] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) 05Open→03Resolved [21:40:22] taavi: are you able to run maintenance scripts in the Beta Cluster? [21:43:09] hauskatze: yes [21:43:27] taavi: could you check https://phabricator.wikimedia.org/T299823#7652496 ? [21:44:32] sure [21:45:05] !log running recountCategories.php on all beta wikis per T299823#7652496 [21:45:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:45:08] T299823: Regularly run recountCategories.php on Wikimedia wikis via systemd timer - https://phabricator.wikimedia.org/T299823 [21:45:40] Kiitoksia oikein paljon [21:46:08] hauskatze: https://phabricator.wikimedia.org/P19566 [21:46:21] note that your command was missing a .php at the end [21:47:06] oh, true [21:47:44] the output looks sane [21:48:02] I'll ping legoktm on task so he can check it [21:48:54] ty [21:49:09] or maybe not? :) [21:49:57] I'm doing other stuff rn, the ty was for testing it and pinging me (in advance) :)) [21:50:23] Sounds like the script is working as expected but I'm no expert [22:28:31] PCC looks good on both eqiad and codfw [22:56:27] Project beta-scap-sync-world build #36969: 04FAILURE in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/36969/ [23:05:57] Yippee, build fixed! [23:05:57] Project beta-scap-sync-world build #36970: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/36970/