[00:24:06] (03PS1) 10Legoktm: [CologneBlue] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513920 (https://phabricator.wikimedia.org/T224780) [00:24:08] (03PS1) 10Legoktm: [Nostalgia] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513921 (https://phabricator.wikimedia.org/T224781) [05:08:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [06:48:53] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:55:36] 10Phabricator (Upstream), 10WMSE-Bug-Reporting-and-Translation-2019, 10Upstream: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10Sebastian_Berlin-WMSE) >>! In T220585#5223972, @epriestley wrote: > I think about 95% of the time you want to move to... [07:05:25] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) [08:03:25] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) p:05Triage→03High [08:16:01] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) A smart script would work as follows: - It first checks its own state. If the server is depooled, restart immediately, return - If th... [08:32:37] I'd like some advice for how to resolve a multiple test failure. In this case, AdvancedSearch tests have gotten stale and are broken in multiple ways due to changes in other repos. We can't merge any single test fix, since the CI jobs will still fail. [08:33:11] We can consider a force-push, but is there anything nicer we might be able to do? [08:33:50] At the least, I'd like to stack all the test fixes on one feature branch to demonstrate that they will solve the failures when taken together. [08:33:59] (before force-pushing.) [08:34:46] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) >>! In T223102#5222173, @hdothiduc wrote: > For the Phab Herald rule, should I simply create... [08:34:56] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) >>! In T223102#5222644, @mcruzWMF wrote: > @Aklapper any news on this? ^^ I currently don't... [08:36:35] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) Community Comms requests form created, based on template linked from https://phabricator.wik... [09:31:01] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for @Urbanecm - https://phabricator.wikimedia.org/T224865 (10Urbanecm) [10:28:48] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10WMDE-leszek) Hello @akosiaris, this is a friendly ping from your favourite WMDE customers :) Any news on the deployment front?... [10:29:06] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:36:10] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:37:55] 10Continuous-Integration-Config, 10phan: Phan-docker output (at least) double-escapes HTML - https://phabricator.wikimedia.org/T224821 (10hashar) [10:46:30] 10Continuous-Integration-Config, 10phan: Phan-docker output (at least) double-escapes HTML - https://phabricator.wikimedia.org/T224821 (10hashar) The phan `-m checkstyle` options is to make it generate a checkstyle report to stdout which is a XML document. That is why some characters end up being escaped. I... [10:51:47] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10ArielGlenn) The 'smart script' outlined above seems fine to me for now. I suppose that on mwdebug we would run a 'dumb script' that would... [11:03:46] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5229810, @ArielGlenn wrote: > Before we start redefining how SWATs are done, let's look at how many errors are caught... [11:05:52] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10ArielGlenn) We'll also have the headache of cli-based long-running jobs and their opcache, which would not be flushed by a php-fpm restart. [11:07:59] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5229851, @ArielGlenn wrote: > We'll also have the headache of cli-based long-running jobs and their opcache, which woul... [11:16:43] 10Continuous-Integration-Config, 10phan: Phan-docker output (at least) double-escapes HTML - https://phabricator.wikimedia.org/T224821 (10Daimona) @hashar Thanks for the explanation! Indeed, the file name is not that explicative... Anyway, a couple of remarks: 1 - The output is actually triple-escaped, since f... [11:23:34] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10MoritzMuehlenhoff) > If the server is pooled, wait for a lock on poolcounter (we can tune appropriately the concurrency allowed) And proba... [11:54:04] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10jijiki) [12:11:39] hashar: Let me know if I can do anything to make this patch chain more palatable, https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/511749/ [12:24:13] awight: you have been productive :] [12:24:53] awight: it is probably fine as is do not worry :] [12:25:09] I am busy migraitng the wikimedia/fundraising/civicrm jobs to Docker this week [12:25:14] got raised during scrum of scrum [12:25:18] ;\ [12:27:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10MarkAHershbe... [12:29:17] anyone else having issues fetching from Gerrit? [12:29:24] SSH works, git-upload-pack hangs [12:29:30] hashar: I can lend a hand with that, if you need anyone to work with in CEST... [12:30:51] I think I was involved in making that whole mess in the first place. [12:31:16] (correction, it doesn’t hang completely, it just takes a very long time) [12:38:26] awight: ah yeah that is true. I am doing a first pass at attempting to migrate it then will come back with a few questons I am sure [12:39:08] Lucas_WMDE: maybe a slow connection on your side? :( [12:39:36] hashar: Okay, thanks for doing that. Looking forward to review! [12:39:43] Lucas_WMDE: not that on each pull/fetch, the Gerrit server sends a copy of every single references it has (eg all of ref/changes/xx/yyyxx/z which are all patchsets made for that repo) and that is a lot of data :\ [12:40:15] 10Gerrit: Fetching from Gerrit is slow or broken - https://phabricator.wikimedia.org/T224878 (10Lucas_Werkmeister_WMDE) [12:40:50] hashar: connection should be reasonably fast [12:41:04] though so far I only have confirmation on this issue from other WMDE colleagues, who would be on the same connection [12:41:23] hmm [12:41:28] * hashar does a few traceroutes [12:41:56] connecting to my own server, I get decent downstream speeds (~40 MiB/s) [12:42:08] I see your connection / ip [12:42:14] Lucas_WMDE: I'm in the WMDE office and have been fine fetching and submitting patches. [12:42:26] and there seems to be some additional latency on the way from Gerrit to that .de IP which I am assuming is the wmde office [12:42:27] Could it be a specific repo? [12:42:35] but that does not seem too bad [12:42:49] I’ve started another git fetch now [12:42:55] awight: so far I’ve seen this in core and Wikibase [12:42:58] haven’t tried other repos yet [12:43:16] hmm core was fine for me. "git fetch --all" sort of thing? [12:43:58] Lucas_WMDE: if all fail, you can try running git fetch with the magic debug mode: GIT_TRACE=1 GIT_TRACE_PACKET=1 git fetch [12:44:03] which will dump all of its activity [12:44:12] it might just be transfering a lot of data :/ [12:44:17] the long story is https://phabricator.wikimedia.org/T103990 [12:44:34] and the fix is to switch to git protocol v2 which our Gerrit does not support yet (gotta upgrade it) [12:45:12] weird, I’ve never noticed it being this slow before [12:45:58] tried the magic debug mode – now it’s transferring lots of refs, but before that it was also hanging for a while [12:45:58] and maybe the path from wikimedia office to gerrit has some congestion at some point. You can try traceroute gerrit.wikimedia.org [12:46:11] I guess the server might have needed some time to collect the refs before it started printing debug output [12:47:15] yeah it takes a bit of time to process all references [12:47:17] :\ [12:47:49] time git fetch gerrit -> 6,2s of wall time [12:47:49] yeah by now it definitely seems to be throttled by sending all the refs [12:48:06] at least I can estimate how long it’ll take – currently at /79 ^^ [13:18:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:24:54] (03CR) 10Umherirrender: [C: 03+1] [Nostalgia] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513921 (https://phabricator.wikimedia.org/T224781) (owner: 10Legoktm) [13:25:01] (03CR) 10Umherirrender: [C: 03+1] [CologneBlue] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513920 (https://phabricator.wikimedia.org/T224780) (owner: 10Legoktm) [13:30:49] (03CR) 10Hashar: [C: 03+2] [Nostalgia] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513921 (https://phabricator.wikimedia.org/T224781) (owner: 10Legoktm) [13:30:53] (03CR) 10Hashar: [C: 03+2] [CologneBlue] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513920 (https://phabricator.wikimedia.org/T224780) (owner: 10Legoktm) [13:31:30] Does fetching take longer then 11mins? [13:32:22] it was more around 5 minutes [13:32:32] (03Merged) 10jenkins-bot: [CologneBlue] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513920 (https://phabricator.wikimedia.org/T224780) (owner: 10Legoktm) [13:32:36] (03Merged) 10jenkins-bot: [Nostalgia] Enable phan [integration/config] - 10https://gerrit.wikimedia.org/r/513921 (https://phabricator.wikimedia.org/T224781) (owner: 10Legoktm) [13:32:39] but no sharp timeout, some didn’t take that long [13:32:43] ok. I guess that was for mw core? [13:32:54] core and Wikibase [13:59:51] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5228040, @akosiaris wrote: >>>! In T220401#5226623, @Eevans wrote: >>>>! In T220401#5226531, @... [14:05:57] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) >>! In T220401#5230202, @Eevans wrote: >>>! In T220401#5228040, @akosiaris wrote: >>>>! In T220401#5226623... [14:14:12] hashar: I'm here whenever you are. [14:14:35] ooo [14:14:58] James_F: joined :) [14:23:29] (03PS1) 10Hashar: (DO NOT SUBMIT) wip civicrm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/514011 [14:25:28] (03PS1) 10Jforrester: Temporarily don't run selenium tests with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) [14:26:58] (03CR) 10jerkins-bot: [V: 04-1] Temporarily don't run selenium tests with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [14:27:49] excuse me for parachuting in here but, this is 'ready to submit' (but prolly needs rebase or something at this point), https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/512672/ [14:27:57] should I... rebase and then poke someone to click something? [14:29:56] (03PS2) 10Jforrester: Temporarily don't run selenium tests with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) [14:32:12] (03CR) 10jerkins-bot: [V: 04-1] Temporarily don't run selenium tests with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [14:33:53] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:05] (03CR) 10Kosta Harlan: Add sonar CodeHealth checks to Wikibase repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [14:35:55] (03CR) 10Kosta Harlan: [C: 03+1] "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [14:38:17] !log hashar and I are temporarily disabling running selenium tests in CI. See T211784 T222406 for more details. [14:38:27] (03Abandoned) 10Hashar: Update Quibble containers to Node 10 with npm 6 [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [14:39:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:39:17] T222406: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 [14:39:29] T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 [14:43:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47368 bytes in 2.621 second response time [14:44:07] (03CR) 10Awight: [C: 04-1] "--skip values should be comma-separated" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [14:44:09] (03PS1) 10Hashar: docker: delete quibble-jessie-hhvm and php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514015 [14:44:51] (03CR) 10WMDE-leszek: "looks good from WMDE, let's see what we learn from it!" [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [14:44:55] (03CR) 10WMDE-leszek: [C: 03+1] Add sonar CodeHealth checks to Wikibase repos [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [14:45:12] (03PS2) 10Hashar: docker: delete quibble-jessie-hhvm and php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514015 [14:45:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:42] (03CR) 10Jforrester: [C: 03+2] docker: delete quibble-jessie-hhvm and php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514015 (owner: 10Hashar) [14:47:28] (03PS1) 10Kosta Harlan: Run mwext-phpunit-coverage-docker-publish in codehealth postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/514016 [14:47:44] (03Abandoned) 10Jforrester: Temporarily don't run selenium tests with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/514012 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [14:48:01] (03Merged) 10jenkins-bot: docker: delete quibble-jessie-hhvm and php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514015 (owner: 10Hashar) [14:50:04] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36457 bytes in 0.873 second response time [14:52:04] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5229898, @MoritzMuehlenhoff wrote: >> If the server is pooled, wait for a lock on poolcounter (we can... [14:52:26] 10Gerrit: Fetching from Gerrit is slow or broken - https://phabricator.wikimedia.org/T224878 (10greg) `lang=irc 05:44:17 <+hashar> the long story is https://phabricator.wikimedia.org/T103990 05:44:34 <+hashar> and the fix is to switch to git protocol v2 which our Gerrit does not support yet (gotta upgrade it) 05... [14:52:54] (03CR) 10Kosta Harlan: [C: 03+1] Add sonar CodeHealth checks to Wikibase repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [14:54:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [15:04:09] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 47356 bytes in 0.912 second response time [15:04:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47356 bytes in 0.743 second response time [15:12:16] (03PS1) 10Hashar: docker: switch quibble-stretch to node10/npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/514021 (https://phabricator.wikimedia.org/T222406) [15:13:32] (03CR) 10Jforrester: docker: switch quibble-stretch to node10/npm6 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/514021 (https://phabricator.wikimedia.org/T222406) (owner: 10Hashar) [15:14:14] (03PS2) 10Hashar: docker: switch quibble-stretch to node10/npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/514021 (https://phabricator.wikimedia.org/T222406) [15:16:31] (03CR) 10Hashar: [C: 03+2] docker: switch quibble-stretch to node10/npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/514021 (https://phabricator.wikimedia.org/T222406) (owner: 10Hashar) [15:18:26] (03Merged) 10jenkins-bot: docker: switch quibble-stretch to node10/npm6 [integration/config] - 10https://gerrit.wikimedia.org/r/514021 (https://phabricator.wikimedia.org/T222406) (owner: 10Hashar) [15:21:31] The following packages have unmet dependencies: (image.py:179) [15:21:31] 2019-06-03 17:17:39,945 [docker-pkg-build] INFO - nodejs-legacy : Depends: nodejs (>= 0.6.19~dfsg1-3~) but it is not going to be installed (image.py:179) [15:21:31] 2019-06-03 17:17:39,953 [docker-pkg-build] INFO - E: Unable to correct problems, you have held broken packages. [15:22:33] Fun. Let's drop nodejs-legacy. [15:25:49] (03PS6) 10Thcipriani: clear_message_blobs: call once [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) [15:27:47] (03PS1) 10Hashar: docker: nodejs10 does provide /usr/bin/node [integration/config] - 10https://gerrit.wikimedia.org/r/514026 (https://phabricator.wikimedia.org/T222406) [15:29:18] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Fix Install Script For Some Versions of Ubuntu - https://phabricator.wikimedia.org/T223715 (10jeena) Yes, I think we could possibly remove docker from the equation. [15:34:44] (03PS1) 10Jforrester: [DNM] Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 [15:34:46] (03PS1) 10Jforrester: Stop defining PHP55 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514032 [15:34:48] (03PS1) 10Jforrester: Drop all PHP55 docker files [integration/config] - 10https://gerrit.wikimedia.org/r/514033 [15:34:56] (03PS1) 10Hashar: Switch most Quibble jobs to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514034 (https://phabricator.wikimedia.org/T222406) [15:35:44] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for @Urbanecm - https://phabricator.wikimedia.org/T224865 (10Krenair) 05Open→03Resolved a:03Krenair [15:37:14] (03CR) 10jerkins-bot: [V: 04-1] Drop all PHP55 docker files [integration/config] - 10https://gerrit.wikimedia.org/r/514033 (owner: 10Jforrester) [15:37:16] (03CR) 10jerkins-bot: [V: 04-1] Stop defining PHP55 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514032 (owner: 10Jforrester) [15:39:05] (03PS2) 10Jforrester: Stop defining PHP55 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514032 [15:39:07] (03PS2) 10Jforrester: Drop all PHP55 docker files [integration/config] - 10https://gerrit.wikimedia.org/r/514033 [15:40:03] (03CR) 10Thcipriani: [C: 03+2] clear_message_blobs: call once [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [15:41:01] (03CR) 10Jforrester: [C: 04-2] "Waiting for the final 1.27 release." [integration/config] - 10https://gerrit.wikimedia.org/r/514031 (owner: 10Jforrester) [15:41:20] (03Merged) 10jenkins-bot: clear_message_blobs: call once [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [15:42:01] (03CR) 10jenkins-bot: clear_message_blobs: call once [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [15:42:49] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for @Urbanecm - https://phabricator.wikimedia.org/T224865 (10Urbanecm) Thank you, @Krenair. [15:46:33] !log reduce number of executors on all integration slave docker, they are somehow starving on CPU and/or IO when lot of mediawiki builds are running in parallel [15:46:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:50:56] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Fix Install Script For Some Versions of Ubuntu - https://phabricator.wikimedia.org/T223715 (10brennen) Cool. I'll push a patch for that shortly. [15:56:34] 10Beta-Cluster-Infrastructure, 10MediaWiki-Configuration: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10WMDE-leszek) [15:57:44] !log Deleting integration-slave-docker-1055 and integration-slave-docker-1056 . CPU is way too slow T223971 [15:58:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:58:14] T223971: Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 [16:00:03] 10Phabricator: Change my Phabricator username from "E to Pi times i" to "Retro" - https://phabricator.wikimedia.org/T224900 (10E_to_the_Pi_times_i) [16:01:54] 10Phabricator: Change my Phabricator username from "E to Pi times i" to "Retro" - https://phabricator.wikimedia.org/T224900 (10E_to_the_Pi_times_i) [16:05:27] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10mmodell) >>... [16:05:42] 10Phabricator: Change my Phabricator username from "E to Pi times i" to "Retro" - https://phabricator.wikimedia.org/T224900 (10E_to_the_Pi_times_i) p:05Triage→03Low [16:06:45] (03PS2) 10Jforrester: [DNM] Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 [16:06:47] (03PS3) 10Jforrester: Stop defining PHP55 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514032 [16:06:49] (03PS3) 10Jforrester: Drop all PHP55 docker files [integration/config] - 10https://gerrit.wikimedia.org/r/514033 [16:06:51] (03PS1) 10Jforrester: wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 [16:06:53] (03PS1) 10Jforrester: wikiba.se: Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 [16:06:57] (03PS1) 10Jforrester: SmashPig, wikimania-scholarships: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 [16:06:59] (03PS1) 10Jforrester: Wikiba.se: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514056 [16:07:01] (03PS1) 10Jforrester: wikimedia-fundraising-crm: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 [16:07:03] (03PS1) 10Jforrester: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 [16:07:05] (03PS1) 10Jforrester: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 [16:10:08] (03CR) 10jerkins-bot: [V: 04-1] SmashPig, wikimania-scholarships: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (owner: 10Jforrester) [16:10:10] (03CR) 10jerkins-bot: [V: 04-1] wikiba.se: Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [16:10:14] 10Gerrit: Fetching from Gerrit is slow or broken - https://phabricator.wikimedia.org/T224878 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved {T103990} is definitely part of this, yes, and it looks like there was some other network congestion on a part of the path from the WMDE office to Gerrit – forcing an SSH... [16:14:05] got a -1 from jenkins, can I assume it's related to the node stuff? https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/512672/ and https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/112379/console [16:15:34] apergos: Yes, CI is expecting https://gerrit.wikimedia.org/r/c/mediawiki/core/+/508692 to have landed but gate is two hours backlogged due to a Minerva patch. [16:15:42] ouch [16:15:48] Tell me about it. :-( [16:15:55] Sorry for the disruption. [16:16:16] it's ok, I just want it to land sometime today (bugfix) so it makes the branch cut tomorrow [16:21:49] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Lucas_Werkmeister_WMDE) > Git protocol 2 support has been added tentatively in Gerrit 2.16 but eventually got withdrawn with 2.16.2: Is there any information on whethe... [16:22:11] 10Phabricator: Change my Phabricator username from "E to Pi times i" to "Retro" - https://phabricator.wikimedia.org/T224900 (10Aklapper) 05Open→03Resolved a:03Aklapper [16:23:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Parsoid, 10phan-taint-check-plugin, and 2 others: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10greg) a:03hashar [16:23:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10greg) a:03hashar Antoine is working on this now. [16:28:05] 10Continuous-Integration-Config, 10Wikidata: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [16:28:56] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Paladox) Gerrit still supports Git protocol 2 (they just disabled it by default rather then enabled it by default). So we can enable once we upgrade. [16:29:00] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:29:28] 10Continuous-Integration-Config, 10Wikidata: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [16:29:30] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:29:50] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:29:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10Jdforrester-WMF) [16:29:56] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:29:58] 10Continuous-Integration-Config, 10Wikidata: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [16:30:08] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:30:10] 10Continuous-Integration-Config, 10Wikidata: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) [16:30:20] 10Continuous-Integration-Infrastructure: Drop php55 testing support - https://phabricator.wikimedia.org/T224907 (10Jdforrester-WMF) [16:31:07] 10Continuous-Integration-Infrastructure: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10Jdforrester-WMF) [16:31:24] 10Continuous-Integration-Infrastructure: Drop php55 testing support - https://phabricator.wikimedia.org/T224907 (10Jdforrester-WMF) [16:31:26] 10Continuous-Integration-Infrastructure: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) [16:31:28] 10Continuous-Integration-Infrastructure: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10Jdforrester-WMF) [16:31:38] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Paladox) See https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/473643/ [16:32:51] hashar: T224908 is just to make you happy. ;-) [16:32:51] T224908: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 [16:32:53] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5230212, @akosiaris wrote: >>>! In T220401#5230202, @Eevans wrote: >>>>! In T220401#5228040, @... [16:33:36] watching jenkins backlog vs watching the grass grow. which is more mind-numbing? [16:37:18] Indeed. :-( [16:42:02] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) 05Resolved→03Open a:05hashar→03None I am not sure what I have done a month ago, but on a new instance it is no more upgrad... [16:43:26] (03PS1) 10Jkroll: [WiP] Add FileImporter to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) [16:50:14] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) I have deleted the affected instances and created two new ones hoping for them to be scheduled on cloud... [16:52:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Parsoid, 10phan-taint-check-plugin, and 2 others: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10greg) p:05Triage→03Normal [16:55:48] James_F: so the containers are published now ;] [16:56:05] "just" have to switch the actual jobs [16:57:50] "Just". [16:57:53] Shall we do it? [16:58:09] I can't remember how I did it last week :/ [16:58:42] probably just updated just a few of them, washed, ,updated few mores etc [16:58:49] or we can just do it and !log it [16:59:10] Do it in one. [17:05:23] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10hashar) wikiba.se got imported from Github to Gerrit via T171160. Back in 2017 wikiba.se was apparently intended to run on a Jessie server and php5.6 ( d0... [17:06:14] James_F: so on ecan update the 143 jobs or so via I24e998fe6f1a936649ae43740f16217fbcf6f1a4 [17:06:25] going to have dinner + take care of kids etc. I will be back in a couple hours [17:06:41] and if nobody di the update, I will and deal with the potential aftermath / thunder of doom etc [17:06:42] ;D [17:08:08] 10Continuous-Integration-Infrastructure: Drop php55 testing support - https://phabricator.wikimedia.org/T224907 (10Jdforrester-WMF) [17:12:42] (03PS3) 10Jforrester: [DNM] Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 (https://phabricator.wikimedia.org/T224907) [17:12:44] (03PS4) 10Jforrester: Stop defining PHP55 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514032 (https://phabricator.wikimedia.org/T224907) [17:12:46] (03PS4) 10Jforrester: Drop all PHP55 docker files [integration/config] - 10https://gerrit.wikimedia.org/r/514033 (https://phabricator.wikimedia.org/T224907) [17:12:48] (03PS2) 10Jforrester: SmashPig, wikimania-scholarships: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) [17:12:50] (03PS2) 10Jforrester: wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 (https://phabricator.wikimedia.org/T223348) [17:12:53] (03PS2) 10Jforrester: Wikiba.se: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514056 (https://phabricator.wikimedia.org/T224905) [17:12:56] (03PS2) 10Jforrester: wikimedia-fundraising-crm: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 (https://phabricator.wikimedia.org/T223348) [17:12:58] (03PS2) 10Jforrester: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [17:13:00] (03PS2) 10Jforrester: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [17:13:02] (03PS1) 10Jforrester: Drop php-compile-hhvm-jessie-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514070 (https://phabricator.wikimedia.org/T224908) [17:13:54] 10Release-Engineering-Team (Kanban), 10Scap, 10MediaWiki-ResourceLoader, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 3 others: Scap deployments are not purging MessageBlobStore (was: Stale localized messages) - https://phabricator.wikimedia.org/T222539 (10thcipriani) Deployed on beta, the old method to... [17:14:21] (03CR) 10jerkins-bot: [V: 04-1] [DNM] Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 (https://phabricator.wikimedia.org/T224907) (owner: 10Jforrester) [17:14:48] (03Abandoned) 10Jforrester: wikiba.se: Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [17:28:21] my change is in, thank you to whoever made the right sacrifice to the jenkins jobs [17:30:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [17:31:09] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops: Deploy scap 3.10.0-1 - https://phabricator.wikimedia.org/T224915 (10thcipriani) [17:32:20] (03CR) 10Jforrester: [C: 03+2] "Let's give this a go." [integration/config] - 10https://gerrit.wikimedia.org/r/514034 (https://phabricator.wikimedia.org/T222406) (owner: 10Hashar) [17:34:43] (03Merged) 10jenkins-bot: Switch most Quibble jobs to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514034 (https://phabricator.wikimedia.org/T222406) (owner: 10Hashar) [17:40:56] !log Reloading Zuul to switch most Quibble jobs to node 10 T222406 [17:40:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:40:59] T222406: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 [17:48:41] (03PS1) 10Thcipriani: Quick and dirty security checker for translations. [integration/config] - 10https://gerrit.wikimedia.org/r/514083 (https://phabricator.wikimedia.org/T205563) [17:50:15] (03CR) 10Jrbranaa: [C: 03+1] Run mwext-phpunit-coverage-docker-publish in codehealth postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/514016 (owner: 10Kosta Harlan) [17:52:11] (03CR) 10Thcipriani: [C: 03+2] Quick and dirty security checker for translations. [integration/config] - 10https://gerrit.wikimedia.org/r/514083 (https://phabricator.wikimedia.org/T205563) (owner: 10Thcipriani) [17:55:10] (03Merged) 10jenkins-bot: Quick and dirty security checker for translations. [integration/config] - 10https://gerrit.wikimedia.org/r/514083 (https://phabricator.wikimedia.org/T205563) (owner: 10Thcipriani) [18:04:06] (03PS1) 10Jforrester: Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) [18:06:53] (03CR) 10jerkins-bot: [V: 04-1] Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [18:08:11] (03PS2) 10Jforrester: Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) [18:10:30] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Jrbranaa) [18:12:47] paladox: Reedy: mystery solved, it was my fucking wifi [18:12:52] heh [18:12:56] I'm at a different university, and just got an email from them [18:13:09] > The NYUIT Office of Information Security has detected some suspicious traffic known as SSH User Authentication Brute Force Attempt . This alert is triggered when there are more than 20 SSH requests in 60 seconds. [18:13:21] Was that you? [18:13:28] did you trigger that? :P [18:13:34] pretty sure yeah [18:13:42] heh [18:13:54] though I'm making outbound ssh connections, which is weird [18:13:58] * Reedy suspects legoktm of controlling a botnet [18:14:41] lol [18:16:45] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: SpamBlacklist - https://phabricator.wikimedia.org/T224921 (10Jrbranaa) [18:18:47] (03CR) 10Jforrester: [C: 03+2] Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [18:21:05] (03CR) 10jerkins-bot: [V: 04-1] Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [18:21:36] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: Collection Extension - https://phabricator.wikimedia.org/T224922 (10Jrbranaa) [18:23:44] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Add Code Health Metrics Working Group members to Code Health mailing list - https://phabricator.wikimedia.org/T205132 (10Jrbranaa) 05Open→03Declined [18:31:39] (03CR) 10Jforrester: [C: 03+1] "Hmm. Not sure what's broken here? Unless it's removal-of-job-use-and-definition-in-one-patch, but I thought that was only a docker <-> jjb" [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [18:31:47] James_F: the change was for jenkins jobs so that needds to be updated by using JJB :] [18:31:52] reloading zuul does not change that! [18:32:22] hashar: For which change? [18:32:46] oh [18:33:09] I guess I got confused :D [18:33:17] oh [18:33:17] https://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ [18:33:19] yeah that one [18:33:20] James_F: ^ [18:33:59] !log hswitch most Quibble jobs to node 10 T222406 - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ T222406 [18:34:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:34:03] T222406: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 [18:34:28] hashar: I did `fab deploy_zuul` for https://gerrit.wikimedia.org/r/514034 . [18:34:33] yeah [18:34:49] so fab deploy_zuul is to reload zuul [18:34:57] and thus apply changes made to zuul/layout.yaml [18:35:11] Oh, but not jjb/mediawiki.yaml. [18:35:13] Duh. [18:35:16] Sorry. [18:35:18] but that change 514034 modifies the Jenkins jobs, that is "unrelated" ;] [18:35:26] no worris [18:36:25] So for that you use… `fab deploy_jjb`? `fab deploy_docker`? [18:37:32] for jjb hmm [18:37:38] no command is available unfortunately [18:37:42] Oh. [18:37:47] So you have to do it manually? [18:37:51] Eww. [18:38:19] OK, so all my testing that everything still worked was for nothing because it wasn't using the new stuff. Oh well. [18:42:26] ;-( [18:42:53] hashar: So, how do you actually deploy it manually? [18:43:19] James_F: ah yeah sorry . So http://www.mediawiki.org/wiki/CI/JJB [18:43:22] has the tutorial [18:43:30] it is actually out of date bah [18:43:41] but should work anyway [18:43:54] Oh, right, I did that way back in the past. [18:44:06] With K.rinkle holding my hand the whole way through. [19:03:51] hashar: Anything I can do to help? [19:05:29] James_F: sorry. I have deployed / refreshed the jobs already! [19:11:03] All looks OK? [19:13:58] James_F: hard to know. But usually I then monitor changes made to mediawiki/.* repos https://gerrit.wikimedia.org/r/#/q/is:open+project:%255Emediawiki/.* [19:14:46] Yeah. [19:15:08] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/514097 is a quick hack to make sure that we've moved things over. :-) [19:15:23] It will fail as we haven't done https://gerrit.wikimedia.org/r/514089 yet. [19:15:53] ;] [19:19:02] Oh, hmm, apparently npm won't warn on mis-matching engines directive unless you're using it as a dependency. *sighs* [19:19:22] Shall we deploy https://gerrit.wikimedia.org/r/514089 then? [19:22:33] James_F: likely ;) [19:22:43] I guess it needs splitting into 3? [19:23:18] 1 to add node10-docker in mediawiki.yaml, 1 to switch over in layout.yaml, and a third to remove node-6 from mediawiki.yaml? [19:23:31] yeah well potentially [19:23:40] but that is fine as a single change [19:23:40] One sec. [19:23:45] Won't it break things? [19:23:47] James_F: so lets just deploy that ;) [19:23:51] OK. [19:23:54] Should I try it? [19:24:10] the change fails on gate and submit most probably because mwgate-node10-docker has not been created in jenkins [19:24:17] Right. [19:24:46] I've got it checkout out in jjb/config locally. [19:25:20] do you want to create the job via jjb? [19:25:29] or i can just do it [19:25:45] I should do `jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/ 'mwgate-node10-docker'`? [19:26:06] yeah that would generate that job on your local machine [19:26:16] s/test/update/ to actually push it to jenkins [19:26:23] jenkins-jobs: error: unrecognized arguments: mwgate-node10-docker [19:26:49] jenkins-jobs --conf etc/jenkins_jobs.ini update config/jjb mwgate-node10-docker [19:27:07] lol [19:27:14] it works for me :-( [19:27:53] OK, I tried `jenkins-jobs --conf etc/jenkins_jobs.ini test config mwgate-node10-docker`. [19:27:54] jenkins-jobs --conf jenkins_jobs.ini test config/jjb mwgate-node10-docker [19:28:08] Which output something that looks right. [19:28:14] cool [19:28:16] so then "update" [19:28:27] which might work assuming you have the proper credentials in jenkins_jobs.ini [19:28:41] and that your account is in whatever ldap group [19:28:43] INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 1 [19:28:47] Apparently it worked? [19:29:05] !log Generated mwgate-node10-docker and deployed via jjb. [19:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:29:20] So… now it should merge? [19:29:41] yes sir! [19:29:52] (03CR) 10Jforrester: [C: 03+2] "mwgate-node10-docker now exists, thanks jjb. Let's go." [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [19:30:16] (03CR) 10Hashar: [C: 03+1] Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [19:30:26] eventually [19:30:38] someone will delete the old mwgate-npm-node-6-docker job [19:30:38] :-) [19:30:45] I tend to just stick it around in case we revert [19:30:51] * James_F nods. [19:30:59] or would delete the old job after a few days or as part as some routine [19:31:04] Do they get pruned with a semi-automatic script, like old docker images? [19:32:13] (03Merged) 10jenkins-bot: Replace mwgate-npm-node-6-docker with mwgate-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/514089 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [19:32:56] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514089 [19:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:33:28] Still lots of npm-node-6-docker jobs, but the mwgate ones are the 'main' ones. [19:34:13] https://integration.wikimedia.org/ci/job/mwgate-node10-docker/4/console [19:34:15] Tah-dah. [19:34:41] `node --version` => v10.15.2 and `npm --version` => 6.5.0 [19:35:19] hehe [19:35:37] and when you reload [19:35:46] zuul dynamically reconfigure all the projects [19:35:49] Yeah. [19:35:50] and update all the changes [19:36:01] Even ones in-flight? [19:36:02] so the node-6 jobs get cancelled/dropped [19:36:05] Nice. [19:36:10] and new node10 ones replace them! [19:36:22] yeah in flight [19:37:28] Before: [19:37:38] mwgate-npm-node-6-docker https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/112459/console : SUCCESS in 1m 42s [19:37:40] After: [19:38:15] mwgate-node10-docker https://integration.wikimedia.org/ci/job/mwgate-node10-docker/4/console : SUCCESS in 46s [19:38:21] different repos? [19:38:21] That's a nice improvement. ;-) [19:38:25] Same patch. [19:38:34] hmm [19:38:46] (It's cheating; https://gerrit.wikimedia.org/r/c/mediawiki/core/+/483614 adds package-lock.json which makes npm install much much faster.) [19:38:57] ... ;D [19:39:09] But still, I'll take a 60% speed improvement. ;-) [19:39:56] ohhh [19:40:00] and that supports "npm ci" [19:40:02] so yeah: 00:00:16.470 added 957 packages in 5.532s [19:40:04] Yes. [19:40:34] Yup. [19:46:57] hashar: Next step, review and merge https://gerrit.wikimedia.org/r/c/integration/quibble/+/510709 and the mega-stack? [19:47:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [19:50:00] James_F: yeah for quibble, will get to it evetnually this week [19:50:07] well I mean [19:50:18] Good luck. [19:50:22] I guess i will hae to do it tomorrow :D [19:50:37] hashar: Are you going to send an e-mail telling everyone that Selenium tests aren't currently running and to be extra-careful? [19:53:05] James_F: don't we have a patch to bump mediawiki/core to use webdriver.io 4 with fiber@3 which works with nodejs10 ? [19:53:38] hashar: No. I could try to fiddle with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/514085 until it passes? [19:55:00] caues WMDE does rely on browser tests iirc [19:55:04] and others [19:55:19] I guess I mis understood the migration path :( [19:55:39] I wonder wether we can have mocha-framework to be used with fiber3 [19:57:24] There. [19:57:38] zeljkof, hashar: is there any way to tweak MAVEN_OPTS when running {name}-maven-java8-docker jobs on CI? [19:58:26] SMalyshev: not sure what that env variable was for :D [19:58:50] hashar: setting JVM runtime options when running mvn afaik [19:59:46] apparently we can pass extra arguments to maven by using "maven_args" in jjb [20:00:22] hashar: but it's not argument... I am not sure argument will work there [20:00:31] and gehel added some logic to use mvnw (maen wrapper) [20:01:10] from dockerfiles/java8/mvn we seem to ultimately do something like [20:01:21] ./mvnw --batch-mode -gs /settings.xml .... [20:01:59] and the default for .... being "clean package" [20:02:11] so I am not sure where the Java extra arguments have to be passed [20:02:39] hmm... maybe I could hack mvnw then... let me see [20:02:53] (note: settings.xml is the config to have maven cache to be at a fixed path ( $XDG_CACHE_HOME/maven ) [20:02:59] setting env var would be cleaner though [20:03:10] it's not cache it's memory size... [20:03:20] hashar: btw, how much memory does CI container have? [20:03:27] that is one of the java magic -X option isn't it ? [20:03:34] hashar: yep [20:03:35] for the container its illimited afaik [20:03:42] but the instances have 2GB RAM sometime [20:03:59] hmm 2G may be too short for blazegraph tests... [20:04:02] or can have up to 32 GBytes if the jenkins job is tied to the special label "m4executor" [20:04:14] but those 32GBytes are shared with builds happening concurrently on the host [20:04:18] 32G woud be plenty enough I imagine [20:04:26] for blazegraph, I am pretty sure we got it on the big instances [20:04:49] yeah but I'm not sure java defaults allow it to use all of it... [20:05:14] # m4executor label are slaves with 4GBytes memory. Lot of our Java [20:05:14] # repositories require extra memory such as mjolnir or refinery. [20:05:14] node: DebianJessieDocker && m4executor [20:10:51] hashar: Hmmmmmmmm. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/514085 works to re-enable selenium tests, apparently? Let's merge it now… [20:10:57] I am surprised. [20:13:10] James_F: maybe it installs the proper fiber ?;) [20:16:08] hashar: It does, I carefully changed to the lowest version of wdio-sync that uses a node10-compat version of fibers. [20:16:24] But I thought that the new wdio-sync would force wdio v5+; apparently not. [20:17:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [20:18:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) Things we apparently don't need to do: Web... [20:19:18] > fibers@3.1.1 install /workspace/src/node_modules/fibers [20:19:20] James_F: ^^^ [20:19:23] Yup. [20:19:38] so whatever depends on fiber [20:19:40] I *did* check. :-P [20:19:50] Selenium does. [20:19:53] But it works. [20:19:57] mocah-framework > wdio-async > fiber [20:20:00] does not bind fiber [20:20:04] Yeah.  [20:20:08] and it magically get installed at the version that matches nodejs [20:20:13] Not magically. [20:20:24] I changed the version so that it would do that. [20:20:28] :-) [20:21:34] OH [20:21:53] Please merge before someone breaks one of the tests. :-) [20:22:23] +2 ed [20:24:33] ok mvnw thing seems to be working... but not sure if it picks up the options [20:26:13] btw it downloads tons of stuff from maven central each time... I wonder if there's place for some caching there [20:26:34] hashar: does jankins have some provisions for .m2 repo caching? [20:26:50] yeah the settings.xml file I refered to earlier [20:27:04] it configures maven to have a local repo under /cache/maven [20:27:08] which is restored between builds [20:28:21] :-) [20:28:45] hashar: judging from log https://integration.wikimedia.org/ci/job/wikidata-query-blazegraph-maven-java8-docker/52/consoleFull doesn't look like it's working too well... [20:28:59] maybe something is not set up properly [20:29:15] :( [20:29:21] dockerfiles/java8/Dockerfile.template:# maven wrapper does not support XDG_CACHE_HOME - T218099 [20:29:21] T218099: Maven Wrapper does not support XDG_CACHE_HOME - https://phabricator.wikimedia.org/T218099 [20:29:28] can this be related? [20:29:56] 00:00:00.511 Defined: CASTOR_NAMESPACE="wikidata-query-blazegraph/master/wikidata-query-blazegraph-maven-java8-docker" [20:30:04] is the magic path where the cache is stored on the central instance [20:30:17] checking [20:30:26] or this one: [20:30:26] 20:22:49 Syncing... [20:30:27] 20:22:49 rsync: change_dir "/wikidata-query-blazegraph/master/wikidata-query-blazegraph-maven-java8-docker" (in caches) failed: No such file or directory (2) [20:30:27] 20:22:49 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [Receiver=3.1.2] [20:30:33] yeah that is the cache restoration [20:30:44] so there is nothing I guess ;D [20:31:15] oh [20:31:21] yeah I wonder why though? [20:31:35] so the comment is obsolete I think [20:31:52] the task is resolved [20:32:01] but the cache only get populated if the job passes [20:32:19] hmm... ok. But there were passing jobs for this one [20:32:37] yeah ;-( [20:32:47] though most of them fails (that's kinda the thing I am debugging now is why it is the case) [20:32:59] but this one passed: https://integration.wikimedia.org/ci/job/wikidata-query-blazegraph-maven-java8-docker/46/ [20:33:08] sorry I am tired [20:33:27] the cache get populated if the job pass AND has been triggered via a CR+2 or after a change got merged [20:33:45] ohhh [20:33:49] sorry :-( [20:33:50] so https://integration.wikimedia.org/ci/job/wikidata-query-blazegraph-maven-java8-docker/46/ [20:33:51] passed [20:34:10] but got triggered via a new patchset (that is handled by the "test" pipeline) [20:34:31] I usually manual-merge blazegraph things since tests take 2hrs and running them twice for each change is so slow... [20:34:31] and thus at the end of the build, the sub job castor-save-workspace-cache does not save it [20:34:41] but I can try and +2 instead [20:34:50] so I am going to manually rebuild https://integration.wikimedia.org/ci/job/wikidata-query-blazegraph-maven-java8-docker/46/rebuild/parameterized [20:34:57] and just change the pipeline to "postmerge" [20:35:01] and that will trigger the save [20:35:04] ah cool [20:35:23] which honestly is a hack :- [20:35:27] :-] [20:36:06] it'll have to wait until #52 is finished though since they are serialized afaik [20:36:12] but it's ok let's wait [20:36:23] hashar: OK, now we seem to have completed T222406, what CI thing do we next need to fix? :-) [20:36:23] T222406: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 [20:36:37] SMalyshev: maybe we can make them to run concurrently? [20:37:00] also you should be able to run it on your local machine [20:37:14] hashar: if it's ok... these are pretty fat and can take 2 hrs [20:38:07] SMalyshev: a potential alternative would be to move those test to some stronger / dedicated machine [20:38:10] maybe a vm in prod [20:38:19] hashar: do you mean like via docker container? never did that before... [20:38:20] or a stronger/dedidcated vm on top of openstack [20:38:39] I am fine with serialized runs for now, we don't run too many of them for now [20:39:09] and once I've figured out flaky tests, probably would not need them every day even... [20:39:27] right now I am running them more than usual because of trying to eliminate bad tests [20:40:11] hmm [20:40:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [20:40:49] so given you have a checkout of blazegraph somewhere and its owned by your unix user stas [20:40:55] you should be able to do something like: [20:41:15] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforrester-WMF) [20:41:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10Jdforrester-WMF) 05Stalled→03Open No longer stalled; everything relevant in CI is running node 10. [20:41:59] docker run --rm -it -v /path/to/checkout/of/blazegraph:/src -v cache:/cache --user=stas --entrypoint=bash docker-registry.wikimedia.org/releng/java8:0.5.1 [20:42:17] hashar: interesting, let me try it [20:42:19] which should drop you in a shell [20:42:29] and from there we have our custom maven wrapper in /usr/local/maven [20:42:31] err [20:42:35] /usr/local/bin/mvn [20:42:56] which might not even be needed [20:42:59] so maybe just cd /src [20:43:02] mvn clean verify [20:43:09] or with whatever test filtering options [20:43:24] it is a bit of a mess though. The containers run as user nobody, so you can end up with some ownership issues :\ [20:44:43] ok once it's done downloading the internet, I'll try it :) [20:45:50] hashar: which user should I use there? I am using "smalyshev" as my local user, but docker says [20:45:57] docker: Error response from daemon: linux spec user: unable to find user smalyshev: no matching entries in passwd file. [20:46:07] ah yeah [20:46:08] (03PS1) 10Jforrester: [OOUI] Add a node10 runner in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 [20:46:10] stupid docker [20:46:22] sorry [20:46:37] so skip passing --user="smalyshev" [20:46:42] the container will not accept it [20:46:56] there is no uuid mapping :\ [20:47:26] so what I end up with [20:47:39] is on the host: chown -R nobody:nobody /path/to/blazegraph [20:47:40] seems to be running [20:47:46] then -v /path/to/blazegraph:/src [20:47:46] let's see what comes out of it [20:47:51] and run docker run without --user [20:47:57] (03CR) 10jerkins-bot: [V: 04-1] [OOUI] Add a node10 runner in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [20:48:12] generally it should not be needing any write access afaik... but we'll see [20:48:13] docker run --entrypoint=bash is nice [20:48:17] so you can then run mvn [20:48:22] and when it stops, resume [20:48:29] eg mvn -rf [20:49:13] or rerun just the failed tests or whatever [20:49:46] I guess we never properly described how to reproduce a ci build [20:49:54] hashar: thanks, this can be handy! [20:50:26] hashar: for -v cache:/cache where the cache is going? [20:50:31] yes [20:50:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10Patch-For-Review: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10Eileenmcnaughton) Thanks heaps! [20:50:44] (03PS2) 10Jforrester: [OOUI] Add a node10 runner in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 [20:50:55] hmm [20:51:00] yeah but where the directory is created? local work dir? [20:51:06] so in the releng/java8 container there is a settings.xml file: [20:51:11] [20:51:11] ${env.XDG_CACHE_HOME}/maven [20:51:11] [20:51:35] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10jeena) [20:51:37] and /usr/local/bin/mvn inject that to the actual maven command (either /usr/bin/mvn or ./mvnw) [20:51:51] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10jeena) [20:51:58] so yeah maven should write its downloaded stuff in the container under /cache/maven [20:52:08] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Add liveness/readiness checks to services in local-charts - https://phabricator.wikimedia.org/T224933 (10jeena) [20:52:10] and if you volume mount that from the host the cache persist between runs [20:52:30] (03CR) 10jerkins-bot: [V: 04-1] [OOUI] Add a node10 runner in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [20:57:03] hashar: judging by my setup, memory settings are working... so let's see how it finishes [20:57:08] thanks for the help! [20:57:53] SMalyshev: good ! :) [20:58:15] James_F: SMalyshev I am heading off and to bed. If sometihng needed we can follow up via phabricator ;] [20:58:29] hashar: I am good for now, thanks again! [20:58:35] good night [20:58:49] (03PS3) 10Jforrester: [OOUI] Add a node10 jobs in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 [20:58:50] hashar: Sure. [21:01:59] (03CR) 10jerkins-bot: [V: 04-1] [OOUI] Add a node10 jobs in experimental [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [21:04:18] although https://gerrit.wikimedia.org/r/c/integration/config/+/500116 was merged on April 19, is it possible that it was not deployed until weeks later? [21:04:37] if so, is there somewhere I could look to see when that change was deployed? [21:05:50] kostajh: very unlikely [21:06:09] because any later zuul change that came after it in the repo would have also re-deployed it if it were forgotten [21:06:20] legoktm: thx. yeah that makes sense. for context, I've been investigating T224672 [21:06:21] T224672: ORES/ChangesListHooksHandlerTest causing build failures in other repos (e.g. UploadWizard) - https://phabricator.wikimedia.org/T224672 [21:06:42] (03PS2) 10Jforrester: Run mwext-phpunit-coverage-docker-publish in codehealth postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/514016 (owner: 10Kosta Harlan) [21:06:43] kostajh: we usually deploy integration/config right away [21:06:52] though yeah one can forget ( I often do :-((( ) [21:06:53] one explanation for why the ORES test started failing is that WikibaseClient and ORES hadn't been tested together before [21:07:17] It's possible that WikibaseClient changed to work properly. [21:07:19] since locally, the test fails with both of those extensions enabled, and I've tried going back in git history with core, ORES and WikibaseClient to see if there was a working state [21:07:31] Wikibase integration is… rough. [21:07:39] probably you could try looking through jenkins job history to see what repos it was cloning at the time [21:07:51] I think we store 2 weeks of logs [21:07:51] James_F: the readd wdio-mocha / enable selenium test passed through https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514085/ [21:08:06] James_F: so I guess you can close tasks, claim success on wikitech and celebrate! [21:08:27] hashar: Yes! Unfortunately Timo's not around so I'm stacking patches on top of his WIP committing package-lock. [21:08:40] I'll drop a note to wikitech-l. [21:09:00] James_F: but https://phabricator.wikimedia.org/T222406 sounds like a candidate for a fix (was to upgrade quibble jobs to node6/node10 [21:09:19] trying to pull up builds from May 27 but the logs are gone :\ [21:09:26] James_F: there a still a few tihngs that need to be polished up though (such as tthe quibble-fresnel container that should be based on node10 container) [21:14:55] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) What affect have the `opcode_invalidate` calls for specific files via sync-file had? Do we only see this corru... [21:20:26] (03PS1) 10Brennen Bearnes: prereqs: Remove docker installation steps [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514166 (https://phabricator.wikimedia.org/T223715) [21:21:47] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) The old cloudvirt are apparently HP ProLiant DL380p Gen8 (https://wikitech.wikimedia.org/wiki/HP_DL380p... [21:27:36] James_F: eventually WikimediaMediaInfo fails npm with node10 https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseMediaInfo/+/513538/ [21:27:44] err I mean https://integration.wikimedia.org/ci/job/mwgate-node10-docker/39/console [21:28:17] maybe they have a package-lock.json commited in [21:28:29] ah yes they [21:28:30] do [21:28:36] 00:00:09.243 npm ERR! Invalid: lock file's eslint-config-wikimedia@0.11.0 does not satisfy eslint-config-wikimedia@0.12.0 [21:28:36] 00:00:09.244 npm ERR! Invalid: lock file's grunt@1.0.3 does not satisfy grunt@1.0.4 [21:28:37] :] [21:30:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10Jdforrester-WMF) [21:30:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [21:30:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) 05Open→03Resolved [21:31:46] I'll fix. [21:32:07] James_F: and probably some other random repos would fail similarly [21:32:14] Yeah. [21:32:15] E-mail sent. [21:32:16] hopefully it is trivial enough ! [21:33:01] Oh, and of course they have old-fibers, lovely. [21:35:11] (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] fabfile.py: docker_deploy command [releng/dev-images] - 10https://gerrit.wikimedia.org/r/510619 (https://phabricator.wikimedia.org/T223328) (owner: 10Brennen Bearnes) [21:39:03] James_F: oh also if we want to phase out jessie from CI, in Phabricator I guess we can just create a milestone [21:39:24] I previously created one for trusty https://phabricator.wikimedia.org/project/view/2698/ [21:39:27] * hashar gets bold [21:40:35] https://phabricator.wikimedia.org/project/profile/4080/ :D [21:41:04] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Migrate contint* hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [21:41:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10hashar) [21:42:15] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Drop php55 testing support - https://phabricator.wikimedia.org/T224907 (10hashar) [21:44:05] er [21:44:25] aren't all the jobs running on jessie because that's what the docker slaves run? [21:45:23] 10Continuous-Integration-Infrastructure (phase-out-jessie): Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) [21:45:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10hashar) >>! In T212774#5049731, @hashar wrote: > Seems good so far. Thank you very much. I wil... [21:47:13] 10Gerrit: Fetching from Gerrit is slow or broken - https://phabricator.wikimedia.org/T224878 (10hashar) That was definitely a network congestion between Gerrit and the internet. I don't have much details though. Using bast3002 is definitely a nice hack. [21:50:39] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10brennen) > How to change/update configuration Yeah, seems like a big one. In practice it seems like copying and editing a sample values.yam... [21:54:44] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Security, 10Upstream: Integrate MFA into Gerrit - https://phabricator.wikimedia.org/T198813 (10hashar) See also T147864 which is for OAuth but might well be rejected in the end in favor of having a single sign on system that would come with MFA: {T189531}. [22:03:37] hashar: I created a task. [22:03:55] legoktm: Antoine means no jessie CI images. The back-end of CI should migrate, too. [22:04:16] yeah I craeted a project for that [22:04:24] gotcha [22:04:31] contint1001 / contint2001 got a task filled by moritzm recently [22:04:48] Cool. [22:04:53] the WMCS instance indeed have to migrate, but I guess I don't really want to deal with puppet/docker etc [22:05:11] they are simple enough that is probably not a priority [22:05:17] contint1001 is going to be hmm thought [22:05:20] though [22:05:21] well hard [22:05:23] or whatever [22:05:28] hashar: https://phabricator.wikimedia.org/T224908 better than a project? [22:05:39] Eh. [22:05:50] tracking tasks work for me [22:05:57] but I prefer https://phabricator.wikimedia.org/project/view/4080/ [22:06:04] I mean I prefer a workboard [22:06:06] anywa [22:06:07] y [22:06:24] the good news is that phabricator supports both. So we can keep both the tracking task and the board ;] [22:07:30] anywayI should sleep for real now [22:07:32] *wave [22:23:01] Kk. [22:37:27] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforrester-WMF) 05Open→... [22:37:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10Jdforrester-WMF) [22:37:35] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforrester-WMF) [22:42:47] 10Continuous-Integration-Config: Replace all the npm* images with node10* ones - https://phabricator.wikimedia.org/T207087 (10Jdforrester-WMF) [22:47:01] (03PS1) 10Jforrester: dockerfiles: Drop npm-test-mathoid, unused [integration/config] - 10https://gerrit.wikimedia.org/r/514186 (https://phabricator.wikimedia.org/T207087) [22:47:18] (03CR) 10Jforrester: [C: 04-2] "No it's not." [integration/config] - 10https://gerrit.wikimedia.org/r/514186 (https://phabricator.wikimedia.org/T207087) (owner: 10Jforrester) [22:47:40] (03Abandoned) 10Jforrester: dockerfiles: Drop npm-test-mathoid, unused [integration/config] - 10https://gerrit.wikimedia.org/r/514186 (https://phabricator.wikimedia.org/T207087) (owner: 10Jforrester) [22:56:18] (03PS1) 10Jforrester: jjb: Migrate Wikibase custom jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514188 [22:56:20] (03PS1) 10Jforrester: jjb: Migrate PHP code coverage jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514189 [22:56:22] (03PS1) 10Jforrester: jjb: Migrate docker-setup-mwext-for-phan to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514190 [22:56:24] (03PS1) 10Jforrester: [DNM] jjb: Migrate quibble-donationinterface-{branch}-php70-docker to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514191 [23:00:48] (03CR) 10Jforrester: [C: 03+2] "Deployed new wikibase-client-docker and wikibase-repo-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/514188 (owner: 10Jforrester) [23:03:09] (03Merged) 10jenkins-bot: jjb: Migrate Wikibase custom jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514188 (owner: 10Jforrester) [23:10:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) Fixes so far: * {icon check, color=green} [[https://gerrit.wikime... [23:14:15] legoktm: Want me to push out the phan -> php 7.2 change? [23:18:46] James_F: docker-pkg was broken for me so I havent tested it at all [23:19:33] legoktm: I'll test before pushing, obviously. :-) [23:21:08] OK, normally I test it against core and a few larger extensions [23:23:11] I'm still using docker-pkg abe9117bfae8b05b053ee15f6aa4c2660098c575 as the newer versions are broken for me, BTW; might be the same issue? [23:49:56] Eh, this is taking too long. I'll look at it tomorrow if you've not already slung it out. :-)