[00:17:14] 10Deployments, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Patch-For-Review: All extension should link to the deployed Git revision from Special:Version - https://phabricator.wikimedia.org/T155747 (10Krinkle) [02:31:13] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, and 2 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) @arielGlenn I built the whole package successfully and uploaded the sshd bina... [02:31:30] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, and 2 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) a:03mmodell [04:53:21] 10Beta-Cluster-Infrastructure, 10DBA, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks, and 2 others: Drop the `wikimedia_editor_tasks_entity_description_exists` table - https://phabricator.wikimedia.org/T226326 (10Marostegui) [04:56:21] 10Beta-Cluster-Infrastructure, 10DBA, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks, and 2 others: Drop the `wikimedia_editor_tasks_entity_description_exists` table - https://phabricator.wikimedia.org/T226326 (10Marostegui) a:03Marostegui So for now I have renamed the table on db1092 and... [05:00:02] 10Beta-Cluster-Infrastructure, 10DBA, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks, and 3 others: Drop the `wikimedia_editor_tasks_entity_description_exists` table - https://phabricator.wikimedia.org/T226326 (10Marostegui) Same has been done on testwikidatawiki on s3: ` root@db1123.eqiad.w... [05:55:59] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, and 2 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10ArielGlenn) >>! In T224677#5277070, @mmodell wrote: > @arielGlenn I built the whole pa... [06:09:43] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10serviceops: Our docker base images lack tags - https://phabricator.wikimedia.org/T218342 (10Joe) 05Open→03Resolved a:03Joe This is already fixed, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/501564 I've just created the new ima... [06:36:17] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [07:00:35] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10serviceops: Our docker base images lack tags - https://phabricator.wikimedia.org/T218342 (10hashar) Indeed https://tools.wmflabs.org/dockerregistry/wikimedia-stretch/tags/ Thank you! [07:06:19] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:55:46] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10hashar) [07:55:48] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Migrate contint* hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [08:25:08] !log Removing mysql-server from the CI slaves # https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517095/ | T225735 [08:25:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:25:10] T225735: Cleanup CI puppet manifests - https://phabricator.wikimedia.org/T225735 [09:01:28] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5274663, @thcipriani wrote: > > Should we be working to implement symlink swapping in scap... [09:09:30] (03PS3) 10Hashar: Delay npm install to selenium/qunit stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/509466 [09:10:12] (03CR) 10jerkins-bot: [V: 04-1] Delay npm install to selenium/qunit stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/509466 (owner: 10Hashar) [09:10:17] (03Abandoned) 10Hashar: Delay npm install to selenium/qunit stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/509466 (owner: 10Hashar) [09:32:24] (03PS2) 10Hashar: [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:33:17] (03CR) 10Hashar: [C: 03+2] "> It is possible to add a test, that extensions starting with BlueSpice running with composer? That needs due to dependency with the main " [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:34:24] (03CR) 10jerkins-bot: [V: 04-1] [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:35:18] (03CR) 10jerkins-bot: [V: 04-1] [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:40:42] (03PS3) 10Hashar: [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:42:46] (03CR) 10Hashar: [C: 03+2] [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:44:18] (03Merged) 10jenkins-bot: [BlueSpiceSubPageTree] Run quibble with composer [integration/config] - 10https://gerrit.wikimedia.org/r/518395 (owner: 10Umherirrender) [09:47:59] (03CR) 10Hashar: [C: 03+2] docker: port zuul-cloner to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/518241 (https://phabricator.wikimedia.org/T224908) (owner: 10Hashar) [09:49:29] (03Merged) 10jenkins-bot: docker: port zuul-cloner to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/518241 (https://phabricator.wikimedia.org/T224908) (owner: 10Hashar) [09:50:44] !log Building container releng/zuul-cloner:0.2.0 and switching jobs # T224908 [09:50:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:50:47] T224908: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 [09:55:22] (03CR) 10Hashar: [C: 03+2] "Deployed. Thank you" [integration/config] - 10https://gerrit.wikimedia.org/r/518297 (owner: 10Jforrester) [09:55:57] (03PS2) 10Hashar: jjb: [docker-zuul-cloner] Point at docker image 0.2.0 based on stretch [integration/config] - 10https://gerrit.wikimedia.org/r/518297 (https://phabricator.wikimedia.org/T224908) (owner: 10Jforrester) [09:56:15] (03CR) 10Hashar: [C: 03+2] jjb: [docker-zuul-cloner] Point at docker image 0.2.0 based on stretch [integration/config] - 10https://gerrit.wikimedia.org/r/518297 (https://phabricator.wikimedia.org/T224908) (owner: 10Jforrester) [09:59:56] (03Merged) 10jenkins-bot: jjb: [docker-zuul-cloner] Point at docker image 0.2.0 based on stretch [integration/config] - 10https://gerrit.wikimedia.org/r/518297 (https://phabricator.wikimedia.org/T224908) (owner: 10Jforrester) [10:01:14] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10hashar) [10:09:48] !log Tag Quibble 0.0.32 @ c0fe6eb # T211701 T218357 T220199 T223752 [10:09:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:09:54] T220199: Quibble: deprecate and phase out EXT_DEPENDENCIES / SKIN_DEPENDENCIES - https://phabricator.wikimedia.org/T220199 [10:09:54] T211701: Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 [10:09:54] T223752: Decouple Quibble planning and execution phases - https://phabricator.wikimedia.org/T223752 [10:09:55] T218357: Quibble space separated options shallow arguments - https://phabricator.wikimedia.org/T218357 [10:15:12] (03PS1) 10Hashar: docker: rebuild for Quibble 0.0.32 [integration/config] - 10https://gerrit.wikimedia.org/r/518660 [10:17:25] (03CR) 10Hashar: [C: 03+2] docker: rebuild for Quibble 0.0.32 [integration/config] - 10https://gerrit.wikimedia.org/r/518660 (owner: 10Hashar) [10:19:24] (03Merged) 10jenkins-bot: docker: rebuild for Quibble 0.0.32 [integration/config] - 10https://gerrit.wikimedia.org/r/518660 (owner: 10Hashar) [10:43:51] (03PS1) 10Hashar: Switch Quibble jobs to 0.0.32 [integration/config] - 10https://gerrit.wikimedia.org/r/518663 [10:56:56] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.85% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:05:36] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [12:00:20] !log Updating Quibble based jobs to use version 0.0.32 [12:00:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:10:41] (03CR) 10Hashar: [C: 03+2] "Updated 193 jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/518663 (owner: 10Hashar) [12:13:02] (03Merged) 10jenkins-bot: Switch Quibble jobs to 0.0.32 [integration/config] - 10https://gerrit.wikimedia.org/r/518663 (owner: 10Hashar) [13:04:33] !log Updating git mirrored repos: integration-cumin:~$ sudo cumin --trace --force 'name:docker' 'find /srv/git -name "*.git" -exec git -C {} fetch --prune \;' [13:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:29:08] hashar: Can you ping me when the Quibble release is live? I just want to want a few console logs... [13:29:46] want to *watch [13:29:51] awight: it is already [13:29:57] Fun times! [13:33:47] Wow, thanks again for all the careful review, it actually seems to be working. [13:42:41] (03PS7) 10Kosta Harlan: Add option for using Apache as server [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) [13:43:52] (03CR) 10jerkins-bot: [V: 04-1] Add option for using Apache as server [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [13:44:13] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10hashar) [13:44:42] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10hashar) p:05Triage→03Normal [13:46:33] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10hashar) p:05Triage→03Normal [13:47:26] (03PS1) 10Kosta Harlan: Codehealth: Skip MW and dependency install for core [integration/config] - 10https://gerrit.wikimedia.org/r/518714 [13:49:55] awight: thanks for the review on the apache patch. I was working on php-fpm integration but will come back to address your comments later this week [13:50:24] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10hashar) Have you had a chance to speak about this //"rebase hell"// during the SRE offsite? I think I answered to all the concerns that have been raised so far. Should I then j... [13:52:56] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10jijiki) [13:59:15] (03CR) 10Kosta Harlan: [C: 04-1] "Maybe my tuning is wrong, but I can't replicate a significant improvement in performance in a CI-like environment. On a DigitalOcean dropl" [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:04:04] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Backlog (Later), and 4 others: MediaWiki web installer do not show extension when their dependency i... - https://phabricator.wikimedia.org/T220514 [14:06:21] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Backlog (Later), and 4 others: MediaWiki web installer do not show extension when their dependency i... - https://phabricator.wikimedia.org/T220514 [14:08:51] hashar: good day, was wondering if there is still a concern about CI resource usage if we wanted to roll out the codehealth pipeline to more extensions? (let's say, 12 more) [14:09:59] 10Continuous-Integration-Infrastructure, 10Core Platform Team Kanban, 10MediaWiki-Installer, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 3 others: install.php --with-extensions silently ignores extensions whose dependencies a... - https://phabricator.wikimedia.org/T225512 [14:10:51] kostajh: for the smallest repos it is probably fine. For anything tha tis high traffic it (core , Wikibase* ) that is probably to cause congestion ;D [14:11:06] then I haven't really looked at the codehealth pipeline progress last week [14:11:25] but [14:11:32] I have removed some jobs last week for wikibase [14:11:41] php71/php70 iic [14:11:45] so that gives some breathing room [14:11:45] k [14:11:56] I guess add them [14:12:09] and we revert if that is too slow/too much usage? [14:18:01] hashar: ok, thanks. we might propose to add a few more [14:18:45] (03CR) 10Kosta Harlan: [C: 04-1] "reconsidering approach" [integration/config] - 10https://gerrit.wikimedia.org/r/518714 (owner: 10Kosta Harlan) [14:19:28] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10Core Platform Team Kanban (Doing), and 2 others: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 (10hashar... [14:20:11] awight: and thanks for the review of the quibble ext dependencies patch last month ( https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/502286/ ) [14:20:37] awight: I actually want to revive that patch and complete it. That would nicely clean up a lot of old stuff in CI [14:22:14] (03CR) 10Hashar: "I will look at creating a job that exercise it. And probably switch it to some tool/utility instead of a test, might be easier to run it t" [integration/config] - 10https://gerrit.wikimedia.org/r/504437 (owner: 10Hashar) [14:22:50] ls di [14:22:55] oops, sorry [14:23:43] Lucas_WMDE: :]]] [14:23:57] I am out for a few, be back later [14:31:54] kostajh: Another factor we should keep in mind is that parallel browser testing can dramatically change the performance requirements for the server. [14:32:27] awight: true. I spent some time looking at that, but don't see a great way to accomplish that with wdio [14:33:08] kostajh: Can we parallelize at the quibble level? [14:33:33] having spent some time over the last week looking at it, and I think Krinkle already said this somewhere, probably the best place to optimize the existing selenium tests is to remove a bunch of extraneous MWBot login calls [14:33:44] awight: that's worth investigating [14:34:29] (03PS2) 10Kosta Harlan: Codehealth: Skip MW install for core [integration/config] - 10https://gerrit.wikimedia.org/r/518714 [14:37:32] (03PS3) 10Kosta Harlan: Codehealth: Skip MW install, generate code coverage from unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/518714 [14:38:31] awight: I'm not really sure how that would work though. Quibble currently contains the apache, php and chromedriver within its container. in the future local-charts world, you could imagine parallelizing quibble + chromedriver making calls to an external container with apache + php [14:39:30] +1 that we should be running multiple containers, to simplify the images if nothing else. Is that already on the roadmap? [14:39:56] on the hazy horizon, AIUI :) [14:40:14] But yeah I hadn't thought about chromedriver as a blocker to parallelization, I think you're right about that. [14:41:29] the other thing that could help us is to use pageLoadStrategy=eager with Chromedriver. it's not yet implemented but someone moved the task to 'in progress' a few days ago :) [14:41:40] that would mean that wdio could start checking elements on the page before all the JS loads [14:42:05] (Firefox currently supports pageLoadStrategy=eager but we moved away from that, so I don't know if there is appetite to return to FF) [14:42:28] but given we have a number of tests that don't actually do anything with JS, this could potentially be a big speed increase for the tests [14:43:31] Sounds like it will be incompatible with some features, but maybe that's a good thing in the end? We should avoid serving FOUC of course. [14:56:50] (03CR) 10WMDE-leszek: "should be ready to go now, see Id0922ad6362a8b657b7036c9aa96ab16303b01ae and I1416b3d1ed68b151d15fd8754a46642f742c0ad3." [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [15:10:04] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5277649, @Joe wrote: >>>! In T224857#5274663, @thcipriani wrote: >> >> Should we be... [15:14:03] (03PS6) 10Jforrester: layout, jjb: [wikiba.se] Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 [15:14:23] (03CR) 10Jforrester: [C: 03+2] "> Patch Set 5:" [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [15:16:39] (03Merged) 10jenkins-bot: layout, jjb: [wikiba.se] Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [15:17:34] !log Reloading Zuul to drop wikiba.se-composer-php56-docker T224905 [15:17:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:37] T224905: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 [15:19:49] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Wikidata, 10wikiba.se website: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Thank you, @WMDE-leszek! [15:19:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [15:20:11] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Wikidata, 10wikiba.se website: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [15:20:17] (03PS1) 10Ejegg: Civi: version-agnostic pkgs for php extns [integration/config] - 10https://gerrit.wikimedia.org/r/518742 [15:20:21] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Wikidata, 10wikiba.se website: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [15:22:05] Hi releng! [15:22:13] Hi ejegg. [15:22:17] Thanks again for the help getting Civi tested on php7+ [15:22:32] Thanks for helping! [15:22:53] It seems to be failing for lack of a the intl extension, and I think this patch might fix it: https://gerrit.wikimedia.org/r/#/c/integration/config/+/518742 [15:23:20] Is there a reason to prefer the php-version-specific names for the extension debian packages? [15:23:40] Oh, interesting. hashar isn't here, and I don't know off-hand. We seem to always used the versioned packages but I don't know why. [15:23:40] the Dockerfile template currently refers to php7.0-intl and php7.0-mysql [15:23:46] * James_F nods. [15:24:01] and the base php version seems to have been bumped to 7.2 [15:24:04] We could at the very least bump to php7.2 patches. [15:24:28] yep, I can change that patch to follow the custom [15:24:50] If you do that I'll merge it and we can work out why it's version-specific later without blocking your team. :-) [15:25:49] (03PS2) 10Ejegg: Civi: bump php version for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/518742 [15:25:53] (03PS3) 10Jforrester: layout: [integration/docroot] Switch from php56 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/516570 [15:25:55] (03PS5) 10Jforrester: layout: SmashPig, wikimania-scholarships: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) [15:25:57] (03PS6) 10Jforrester: layout, jjb: wikimedia-fundraising-crm: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 (https://phabricator.wikimedia.org/T223348) [15:25:59] (03PS6) 10Jforrester: jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [15:26:01] (03PS6) 10Jforrester: dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [15:26:23] oh wow, doing some serious cleanup today, huh? ^^^ [15:26:40] (03CR) 10Awight: Civi: bump php version for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518742 (owner: 10Ejegg) [15:27:46] (03CR) 10Awight: Civi: bump php version for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518742 (owner: 10Ejegg) [15:28:06] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) (owner: 10Jforrester) [15:28:08] ejegg: Not yet; we're blocked (amongst other things) on SmashPig. ;-) [15:28:29] James_F: hmm, awight does have a point about matching the production PHP version. [15:28:41] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) (owner: 10Jforrester) [15:28:46] (03CR) 10Jforrester: Civi: bump php version for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518742 (owner: 10Ejegg) [15:28:46] unfortunately that's 7.0 for now [15:29:20] (03CR) 10jerkins-bot: [V: 04-1] layout, jjb: wikimedia-fundraising-crm: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [15:29:32] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) [15:29:51] 7.0 is fine, but it's still running a php56 job. [15:29:51] James_F: Let me see, I'm pretty sure we're all done with php5.6 machines in the fundraising cluster where SmashPig is used [15:29:57] lemme just confirm [15:30:00] * James_F nods. [15:30:12] Killing php7.0 is the next task after php5.6. ;-) [15:36:39] James_F: I get the feeling that you actually know where the template parameters come from when docker-pkg renders the final Dockerfile... would you mind educating me? [15:37:00] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Release Pipeline, 10serviceops-radar: Pipeline: provide a way to rebuild all blubber images - https://phabricator.wikimedia.org/T214431 (10Joe) [15:37:38] awight: "php" is an actual docker image: https://tools.wmflabs.org/dockerregistry/releng/php/tags/ [15:37:49] It's not a parameter. [15:38:15] Where do those functions come from, e.g. {{ ... | image_tag }} [15:38:21] Because the first set of docker images got created without explicit version numbers in their names, and we never went back and fixed it. [15:38:38] So we have php55, php56, php71, php72, php73… and php. [15:38:42] That's believable :-) [15:39:21] Great--https://doc.wikimedia.org/docker-pkg/#variables [15:39:26] Yeah. [15:39:38] docker-pkg provides the magic. [15:42:08] (03PS7) 10Jforrester: layout, jjb: wikimedia-fundraising-crm: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 (https://phabricator.wikimedia.org/T223348) [15:43:24] James_F: ok, php5.6 is indeed dead in fundraising production environments [15:43:32] \o/ [15:43:39] so it should be fine to kill those tests for SmashPig [15:44:05] awight: heh, I'll save that one till we're on a still-supported 7.x version :) [15:44:48] I know the feeling... [15:50:22] 10Continuous-Integration-Config: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) [15:51:06] 10Continuous-Integration-Config: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) [15:51:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) [15:51:10] 10Release-Engineering-Team (Kanban), 10Security-Team, 10phan-taint-check-plugin: Phan-taint-check-plugin not available for PHP > 7.0 - https://phabricator.wikimedia.org/T207344 (10Jdforrester-WMF) [15:51:15] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [15:51:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) 05Open→03Resolved [15:51:35] (03PS2) 10Jforrester: layout: Drop php70-phan-seccheck entirely [integration/config] - 10https://gerrit.wikimedia.org/r/517749 (https://phabricator.wikimedia.org/T226420) [15:51:43] (03CR) 10jerkins-bot: [V: 04-1] layout: Drop php70-phan-seccheck entirely [integration/config] - 10https://gerrit.wikimedia.org/r/517749 (https://phabricator.wikimedia.org/T226420) (owner: 10Jforrester) [15:52:03] (03PS2) 10Jforrester: jjb: Drop php70-phan-seccheck jobs, unused [integration/config] - 10https://gerrit.wikimedia.org/r/517750 (https://phabricator.wikimedia.org/T226420) [15:52:12] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop php70-phan-seccheck jobs, unused [integration/config] - 10https://gerrit.wikimedia.org/r/517750 (https://phabricator.wikimedia.org/T226420) (owner: 10Jforrester) [15:54:19] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Add/reserve a Jenkins node for the pipeline's trigger jobs - https://phabricator.wikimedia.org/T224069 (10brennen) 05Open→03Resolved [16:00:09] 10Phabricator: Change my Phabricator username - https://phabricator.wikimedia.org/T226423 (10Tassu) [16:03:27] 10Continuous-Integration-Infrastructure, 10Core Platform Team Kanban, 10MediaWiki-Installer, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 3 others: install.php --with-extensions silently ignores extensions whose dependencies a... - https://phabricator.wikimedia.org/T225512 [16:16:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10Pchelolo) [16:28:05] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Add/reserve a Jenkins node for the pipeline's trigger jobs - https://phabricator.wikimedia.org/T224069 (10hashar) I have unmounted `/srv` and removed it from `/etc/fstab` :) [16:35:31] (03PS2) 10Awight: Minor copyediting [integration/quibble] - 10https://gerrit.wikimedia.org/r/510246 [17:11:16] James_F: are you using dch for bumping changelog entries in integration/config? [17:12:15] (03PS7) 10Jforrester: jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [17:12:23] (03PS7) 10Jforrester: dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [17:12:24] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10mmodell) [17:13:42] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10mmodell) = This is now unblocked! 🎉 [17:13:42] kostajh: No, don't have it available on my machine, sadly. [17:13:54] kostajh: It's all manual fakery. [17:14:04] James_F: yeah me neither. So I just copy/paste and make up timestamps [17:14:14] yay [17:15:12] twentyafterfour: what is the plan for wmf.10? [17:15:35] Krinkle: I'd like to deploy wmf.10 soon given that it appears to be unblocked now [17:15:37] I'd like at least a few hours to be able to find issues with wmf.10 on group1 before going to group2 [17:15:47] Krinkle: agreed [17:15:48] and I imagine QA will want that as well [17:16:33] cool, yeah, might be able to pull off within a day. maybe group2 tomorrow early, dont know depends on QA I guess. [17:16:38] I'll deploy to group1 right away and we can go from there. Can go to group2 later this afternoon if everything looks good or we can do group2 tomorrow during the train for wmf.11 [17:16:49] * Krinkle nods [17:30:48] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10mmodell) Given the above, the current plan is as follows: 1. I intend to push 1.36.0-wmf.10 to group 1 today, and then let it bake for a... [17:42:51] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Wikimedia-production-error (Shared Build Failure): Composer\Downloader\TransportException in CI builds for multiple repositories - https://phabricator.wikimedia.org/T226253 (10Jdforrester-WMF) 05Open→03Resolved Yeah, this appears to have fixe... [17:45:39] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10mmodell) [17:55:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10mmodell) [18:28:16] Project beta-scap-eqiad build #255019: 04FAILURE in 0.81 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/255019/ [18:31:15] Project beta-scap-eqiad build #255020: 04STILL FAILING in 0.82 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/255020/ [18:31:38] ottomata: ^ [18:31:44] click on those, look at the console output [18:31:49] OH over here! [18:31:50] great [18:31:58] that's great. [18:33:18] James_F: seeing "package mediawiki/phan-taint-check-plugin at version 1.5.0 has a PHP requirement incompatible with your PHP version (7.2.16)" in a build for MinervaNeue, does that look familiar? [18:33:25] https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/518151#message-2c98aba98eae573480b33984ac04bd3a31df56e0 [18:33:39] Yippee, build fixed! [18:33:39] Project beta-scap-eqiad build #255021: 09FIXED in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/255021/ [18:34:15] ok [18:34:32] > Notice: Undefined variable: wmgMonologAvroSchemas [18:34:59] That's me. should be fixed yes? [18:36:30] ottomata: yep stopped in logstash ~ 4.5min ago [18:36:32] cool [18:36:34] great. [18:36:59] logstash-beta is a pain to get into still with the whole access issue. [18:37:12] wonder what the status is on that. [18:37:21] it requires 3 more authentication steps than prod logstash which is funny [18:37:31] secrets yo [18:40:54] kostajh: Yes, that's why mwskin-php72-phan-seccheck-docker is non-voting everywhere. It's waiting on Security to fix seccheck to work with php7.2 [18:42:18] James_F: Oh, indeed it is non-voting, my bad [18:42:33] RoanKattouw: :-P [18:42:42] You're evil, indeed. [18:43:24] So here's a voting failure: [18:43:25] 10:51:07 gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. [18:44:11] RoanKattouw: Which job? [18:44:21] https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/17626/console [18:44:34] Appears to be triggered by Wikibase from a shallow look at what it's doing right before the failure [18:44:49] RoanKattouw: Which job triggered that? [18:45:05] The patch ( https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/518151 ) also has two other voting failures, but those both look like flaky tests [18:45:08] s/job/commit/ [18:45:39] Oh, not WIkidata-related, sorry [18:45:40] 10:50:58 INFO:quibble.commands:Running "npm test" for MinervaNeue [18:45:49] Yeah. [18:49:13] (03CR) 10Hashar: Clone requirements from ext dependencies (035 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [18:49:40] (03PS4) 10Hashar: Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) [18:49:46] (03CR) 10jerkins-bot: [V: 04-1] Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [18:50:11] From reading package-lock.json (which was added to the repo recently), it looks like wdio-mocha-framework (which you added to the repo recently) depends on fibers, which appears to need python in order to build [18:51:03] RoanKattouw: It shouldn't use that version of fibers at all. [18:51:54] Aha. Well the comming adding the package-lock.json file was merged on Thursday, so that explains why the repo's only recently been broken [18:52:34] Mind if I shoot first and ask questions later, i.e. remove the package-lock.json file? Or is there an easier way to fix this? [18:52:36] It should be using 3.1.1, which is specified in package-lock. [18:52:45] Please don't. [18:52:52] (Also, I would have expected that commit to not have passed CI in the first place) [18:53:16] OK, I'll let you work your magic [18:54:50] Yes, it's a puzzler. [18:57:54] Yeah, it's the use of an old version of wdio-cucumber-framework which is pinning fibers to v2.x which is a problem because CI requires fibers v3.x. [18:57:59] * James_F sighs. [18:58:07] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10Legoktm) >>! In T226240#5273779, @mmodell wrote: > @hashar: we already have a read-only replica of most repositories on p... [19:02:10] RoanKattouw: Filed as https://phabricator.wikimedia.org/T226442 [19:02:22] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) [19:02:40] Thanks [19:03:59] Will poke later. [19:04:01] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) [19:05:00] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) [19:05:21] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) (gah, sorry, fighting my own herald rule) [19:07:40] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10Paladox) We can use gerrit slave feature for this (e.g a readonly). (gerrit2001 being the slave) See https://gerrit.goog... [19:32:00] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10greg) [19:44:31] (03CR) 10Thcipriani: [V: 03+2] Gerrit v2.15.14 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/518039 (owner: 10Thcipriani) [19:57:27] so I'm having trouble understanding how this failed: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/518782/ [19:57:38] it's a cherry pick, and the patch to master merged [19:57:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10LibUp: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Paladox) [19:58:07] (03PS5) 10Hashar: Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) [19:58:08] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/598/console is the failure log [19:58:14] (03CR) 10Awight: [C: 03+1] "I would merge--it just needs a rebase over the recent mess I made." (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [19:58:25] (03CR) 10jerkins-bot: [V: 04-1] Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [20:00:25] (03CR) 10Hashar: [C: 04-1] "PS4 fix a few trivial things" [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [20:00:27] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10LibUp: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Paladox) Gerrit has been patched now, so this is now unblocked! [20:05:43] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Wikimedia-production-error (Shared Build Failure): Composer\Downloader\TransportException in CI builds for multiple repositories - https://phabricator.wikimedia.org/T226253 (10hashar) For setting up a packagist.org mirror there is T203529 [20:16:25] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10hashar) >>! In T226240#5279874, @Legoktm wrote: >>>! In T226240#5273779, @mmodell wrote: >> @hasha... [20:45:21] (03CR) 10Awight: Clone requirements from ext dependencies (038 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [21:10:05] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Implement pipeline config validation - https://phabricator.wikimedia.org/T225335 (10brennen) [[ https://gerrit.wikimedia.org/r/518333 | 518333 ]] checks that the configuration contains a top-level `pipelines`. A next step is turnin... [21:16:09] RoanKattouw, James_F did you solve that 'Can't find Python executable "python", you can set the PYTHON env variable.' error? [21:16:32] Minerva patches also fail to merge because of that ;/ [21:16:46] raynor: Sorry, I have a patch locally but forgot to push before getting distracted. [21:17:55] no worries, I just checked logs and noticed that you talked about this issue [21:19:43] James_F -> I'll just edit the task description and add the error message. I tried to search in phab first but It didn't show up (because it doesn't specify the error message) [21:32:14] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2619 MB (5% inode=53%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [21:47:21] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10LibUp: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10thcipriani) 05Open→03Resolved a:03thcipriani Config change has been deployed. Will announce on Wi... [21:47:32] 10Gerrit: Users with a different name in the cn field compared to uid field cannot use http auth - https://phabricator.wikimedia.org/T225308 (10thcipriani) 05Open→03Stalled >>! In T225308#5243509, @mmodell wrote: > @thcipriani: the fix is turning on http passwords. Done. Setting this task to "Stalled" sin... [21:56:51] raynor: Ready for you to merge. [21:57:50] (03PS1) 10Jforrester: Revert "docker: rm no more used releng/npm-test-graphoid image" [integration/config] - 10https://gerrit.wikimedia.org/r/518826 [21:57:52] (03PS1) 10Jforrester: jjb: Bring back graphoid-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/518827 [21:57:54] (03PS1) 10Jforrester: layout: [graphoid] Restore deploy repo testing [integration/config] - 10https://gerrit.wikimedia.org/r/518828 [21:57:56] (03PS1) 10Jforrester: layout: [change-propagation] Deploy is not archived, just a no-op now [integration/config] - 10https://gerrit.wikimedia.org/r/518829 [21:58:58] (03CR) 10Jforrester: [C: 03+2] "Still needed, sadly." [integration/config] - 10https://gerrit.wikimedia.org/r/518826 (owner: 10Jforrester) [22:00:33] (03Merged) 10jenkins-bot: Revert "docker: rm no more used releng/npm-test-graphoid image" [integration/config] - 10https://gerrit.wikimedia.org/r/518826 (owner: 10Jforrester) [22:01:30] !log Re-publishing npm-test-graphoid, whoops. [22:01:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:01:40] (03CR) 10Jforrester: [C: 03+2] jjb: Bring back graphoid-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/518827 (owner: 10Jforrester) [22:02:07] !log Pushed graphoid-deploy-npm-node-6-docker into jjb [22:02:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:04:02] (03Merged) 10jenkins-bot: jjb: Bring back graphoid-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/518827 (owner: 10Jforrester) [22:04:10] (03CR) 10Jforrester: [C: 03+2] layout: [graphoid] Restore deploy repo testing [integration/config] - 10https://gerrit.wikimedia.org/r/518828 (owner: 10Jforrester) [22:04:20] (03CR) 10Jforrester: [C: 03+2] layout: [change-propagation] Deploy is not archived, just a no-op now [integration/config] - 10https://gerrit.wikimedia.org/r/518829 (owner: 10Jforrester) [22:05:46] (03Merged) 10jenkins-bot: layout: [graphoid] Restore deploy repo testing [integration/config] - 10https://gerrit.wikimedia.org/r/518828 (owner: 10Jforrester) [22:06:04] (03Merged) 10jenkins-bot: layout: [change-propagation] Deploy is not archived, just a no-op now [integration/config] - 10https://gerrit.wikimedia.org/r/518829 (owner: 10Jforrester) [22:08:45] !log Zuul: Restore graphoid deploy repo tests, and comment on change-propagation ones. [22:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:26:08] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2661 MB (5% inode=53%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [23:28:06] * thcipriani cleans contint1001 [23:30:26] RECOVERY - Disk space on contint1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [23:31:16] !log cleaned contint1001 docker images using https://phabricator.wikimedia.org/P8604 [23:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:52:50] Krinkle & reedy: what do you think about pushing wmf.10, should it wait for tomorrow or go ahead tonight? [23:58:02] twentyafterfour: wasn't able to finish my logstash groom today, not something I usually do on Monday. [23:58:25] But I'm OK for pushing forward. From Logstash POV we've found all we can within group1 [23:58:30] assuming the open ones are fixed [23:58:41] https://phabricator.wikimedia.org/T226448 is still unresolved [23:59:00] but affects commons only, which is broken already now given its in wmf.10 [23:59:07] hmm