[00:01:30] addshore: Want to work on https://phabricator.wikimedia.org/T224905 ? ;-) [00:02:54] 10Continuous-Integration-Config, 10phan: phan stopped respecting dependencies - https://phabricator.wikimedia.org/T225524 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF WikibaseLexeme seems fixed. Please open new tickets if there are missing phan dependencies. [00:16:19] (03PS2) 10Jforrester: layout: [WebAuthn] Skip HHVM in extension-quibble [integration/config] - 10https://gerrit.wikimedia.org/r/509529 [00:16:55] (03CR) 10Jforrester: layout: [WebAuthn] Skip HHVM in extension-quibble (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/509529 (owner: 10Jforrester) [00:22:13] (03Abandoned) 10Jforrester: Skip php70 tests for wmf branches; we don't run it in prod [integration/config] - 10https://gerrit.wikimedia.org/r/502343 (owner: 10Jforrester) [00:25:06] (03CR) 10Jforrester: [C: 04-1] Whitelist Ammarpad (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/514233 (owner: 10Urbanecm) [00:26:09] (03PS2) 10Jforrester: Whitelist: Add Ammarpad [integration/config] - 10https://gerrit.wikimedia.org/r/514233 (owner: 10Urbanecm) [00:26:53] (03CR) 10Jforrester: [C: 03+2] [ContentTranslation] Add GlobalPreferences as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/514354 (owner: 10Umherirrender) [00:28:20] (03CR) 10Jforrester: [C: 03+2] Whitelist: Add Ammarpad [integration/config] - 10https://gerrit.wikimedia.org/r/514233 (owner: 10Urbanecm) [00:28:41] (03Merged) 10jenkins-bot: [ContentTranslation] Add GlobalPreferences as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/514354 (owner: 10Umherirrender) [00:30:29] (03Merged) 10jenkins-bot: Whitelist: Add Ammarpad [integration/config] - 10https://gerrit.wikimedia.org/r/514233 (owner: 10Urbanecm) [00:36:03] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:49:05] (03PS1) 10Jforrester: Whitelist: Add WMSE to the regex [integration/config] - 10https://gerrit.wikimedia.org/r/516578 [00:49:07] (03PS1) 10Jforrester: Whitelist: Explicitly list each WMF staffer, move formers to trusted others list [integration/config] - 10https://gerrit.wikimedia.org/r/516579 [00:49:51] !log Updated Zuul phan dependencies for ContentTranslation, adding GlobalPreferences [00:49:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:50:33] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:38:40] (03PS4) 10Kosta Harlan: Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) [04:28:19] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [04:56:30] (03CR) 10Mobrovac: "LGTM, but I'd do it in two steps: first, I'd put these as experimental jobs and move them to voting in a second step. We are not in a hurr" [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [06:48:19] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:31:07] (03CR) 10Nikerabbit: [C: 03+1] "Fine for me." [integration/config] - 10https://gerrit.wikimedia.org/r/516500 (owner: 10Kosta Harlan) [07:37:58] (03CR) 10Hashar: [C: 03+2] Whitelist: Add WMSE to the regex [integration/config] - 10https://gerrit.wikimedia.org/r/516578 (owner: 10Jforrester) [07:39:33] (03Merged) 10jenkins-bot: Whitelist: Add WMSE to the regex [integration/config] - 10https://gerrit.wikimedia.org/r/516578 (owner: 10Jforrester) [07:41:26] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), 10User-Ladsgroup: Improve caching in CI tests - https://phabricator.wikimedia.org/T225496 (10hashar) [08:00:07] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), 10User-Ladsgroup: Improve caching in CI tests - https://phabricator.wikimedia.org/T225496 (10hashar) I am not quite sure how MediaWiki selects the caches its is going to use. A few notes: * for the PHPU... [08:19:22] 10Continuous-Integration-Infrastructure, 10Quibble: Consider httpd for quibble instead of php built-in server - https://phabricator.wikimedia.org/T225218 (10hashar) Would you be able to run the same benchmark for hhvm as well? `hhvm -m server -p 8888 -d "hhvm.static_file.extensions[svg]=image/svg+xml"` Nice r... [09:00:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10WMDE-leszek) [09:04:07] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, and 2 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10LucasWerkmeister) Perhaps you could also switch the sshd config to `AuthorizedKeysFile... [09:05:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10ayounsi) [09:05:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10ayounsi) 05Resolved→03Open > DISK WARNING - free space: / 3588 MB (8% inode=63%): https://icing... [10:02:54] 10Continuous-Integration-Config: Issue with inter-extension-dependencies in CI tests - https://phabricator.wikimedia.org/T224285 (10ItSpiderman) Recently, requirement for ExtJSBase in BlueSpiceFoundation@REL1_31_dev was changed to "~3.1" which should match version "1.34" of ExtJSBase@master, but the tests still... [10:26:27] 10Release-Engineering-Team, 10MediaWiki-extensions-General: Fix flapping tests in gated extensions as of 2019-06-06 - https://phabricator.wikimedia.org/T225162 (10WMDE-leszek) [10:53:20] 10MediaWiki-Releasing, 10Release-Engineering-Team: signatures were invalid: EXPKEYSIG 90E9F83F22250DD7 MediaWiki releases repository - https://phabricator.wikimedia.org/T225601 (10Aklapper) [11:20:14] are there any problems with zuul at the moment? https://gerrit.wikimedia.org/r/516516 is supposed to be in gate-and-submit but I don’t see it on the status page [11:22:12] I’ve had issues in the past where some pipeline entries would be invisible for whatever reason, but IIRC in those cases the counter per pipeline was still accurate, and right now I see “gate-and-submit: 0” [11:23:07] hm, but if I look at the status.json responses in the network tab, I see over a hundred entries in gate-and-submit o_O [11:23:41] but most of them with an empty list of heads, apparently [12:38:31] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: debian signing keyid 90E9F83F22250DD7 has expired - https://phabricator.wikimedia.org/T225624 (10Kghbln) [12:50:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10hashar) [12:57:05] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: debian signing keyid 90E9F83F22250DD7 has expired - https://phabricator.wikimedia.org/T225624 (10RazeSoldier) [13:00:59] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: signatures were invalid: EXPKEYSIG 90E9F83F22250DD7 MediaWiki releases repository - https://phabricator.wikimedia.org/T225601 (10Kghbln) [13:01:44] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: signatures were invalid: EXPKEYSIG 90E9F83F22250DD7 MediaWiki releases repository - https://phabricator.wikimedia.org/T225601 (10Kghbln) Added projects as done in T141400 [13:02:32] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: signatures were invalid: EXPKEYSIG 90E9F83F22250DD7 MediaWiki releases repository - https://phabricator.wikimedia.org/T225601 (10Legoktm) https://wikitech.wikimedia.org/wiki/Releases.wikimedia.org#G... [13:02:48] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: debian signing keyid 90E9F83F22250DD7 has expired - https://phabricator.wikimedia.org/T225624 (10Kghbln) @RazeSoldier Thanks. Did not see that. [13:09:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10hashar) We really need to phase out the list of dependencies being maintained in `integration/config.git`. Adding phan dependencies... [13:11:21] 10Continuous-Integration-Infrastructure, 10Zuul: Jenkins-bot should warn that a patch set has changes it depends on that haven't been merged yet - https://phabricator.wikimedia.org/T225495 (10hashar) [13:21:36] 10Continuous-Integration-Infrastructure, 10Zuul: Jenkins-bot should warn that a patch set has changes it depends on that haven't been merged yet - https://phabricator.wikimedia.org/T225495 (10hashar) 05Open→03Stalled p:05Triage→03Low That is for #zuul. It does log debug message for the action it takes,... [13:29:55] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5247276, @thcipriani wrote: >>>! In T224857#5243133, @Joe wrote: >> Regarding rollbacks: I guess that... [13:30:27] 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 - https://phabricator.wikimedia.org/T225628 (10hashar) [13:33:32] 10Continuous-Integration-Config, 10Release-Engineering-Team: Remove PHP_BIN from WMF CI job configuration - https://phabricator.wikimedia.org/T225451 (10hashar) We might still have some old legacy jobs that could still be relying on the php wrapper (the migration was/is tracked by #ci-slipway). It is in `integ... [13:33:48] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Technical-Debt: Remove PHP_BIN from WMF CI job configuration - https://phabricator.wikimedia.org/T225451 (10hashar) [13:45:54] 10Project-Admins: Project for VVIT WikiConnect's Annual Plan (2019-2020) - https://phabricator.wikimedia.org/T225392 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #vvit_wikiconnect_s_annual_plan_2019-2020 has been created: https://phabricator.wikimedia.org/project/view/4102/ Please... [14:05:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10zeljkofilipin) This task was mentioned in [[ https://www.me... [14:08:33] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10fgiunchedi) Could you try running the upgrade twice ? But yes if the jessie image is updated then it should come with the latest rsyslog I... [14:09:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10zeljkofilipin) I think I've found it. This? https://integr... [14:10:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10Michael) >>! In T224899#5253379, @zeljkofilipin wrote: > Th... [14:10:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10zeljkofilipin) p:05Triage→03Normal [14:15:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Configuration, 10Patch-For-Review, 10User-zeljkofilipin: Fatal error Cannot redeclare wmfLabsSettings() on Beta cluster wikis - https://phabricator.wikimedia.org/T224899 (10zeljkofilipin) Now that I know what to look for, I've check... [14:19:31] 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 - https://phabricator.wikimedia.org/T225628 (10Legoktm) This would effectively just kill the LTS effort if we can't adequately test releases and force people who care into using proprietary CI systems. Looking at PHP's EOL i... [14:30:11] (03CR) 10Mholloway: "Thanks, Marko. So just to be clear, the only danger to doing this now is that some Docker images will be published and likely never used?" [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [14:38:46] 10Continuous-Integration-Infrastructure, 10Quibble: Consider httpd for quibble instead of php built-in server - https://phabricator.wikimedia.org/T225218 (10kostajh) > Would you be able to run the same benchmark for hhvm as well? I don't have hhvm set up on my machine, but if I get it configured, then sure :)... [14:40:34] (03PS1) 10Mholloway: Migrate recommendation-api to the service pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/516635 [14:46:40] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) >>! In T225112#5253208, @hashar wrote: > [snip] That's exactly what @Umherirrender proposed a few comments above... (T22511... [14:58:39] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10Core Platform Team Kanban (Doing), and 2 others: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 (10Legokt... [15:12:02] (03CR) 10Mobrovac: "> Thanks, Marko. So just to be clear, the only danger to doing this" [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [15:30:35] 10Continuous-Integration-Infrastructure: CI monitoring to detect flapping tests, especially in unrelated gated extensions - https://phabricator.wikimedia.org/T225193 (10awight) The "recheck bot" is an interesting twist--I like the idea of automatically rechecking if there was an external error e.g. network glitc... [15:40:55] (03CR) 10Hashar: "We also need to get rid of the Jenkins jobs. They are defined in the file jjb/mediawiki-services.yaml with the block:" [integration/config] - 10https://gerrit.wikimedia.org/r/516635 (owner: 10Mholloway) [15:43:38] (03CR) 10Hashar: Migrate mobileapps to the service pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [15:59:39] 10Continuous-Integration-Infrastructure, 10Quibble: Consider httpd for quibble instead of php built-in server - https://phabricator.wikimedia.org/T225218 (10hashar) >>! In T225218#5253448, @kostajh wrote: > I started experimenting with this; I started with Apache then abandoned it because of the difficulties o... [16:00:45] 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 - https://phabricator.wikimedia.org/T225628 (10Jdforrester-WMF) >>! In T225628#5253393, @Legoktm wrote: > I think we can handle that, especially in the new docker based setup. Which "we" are you expecting to do the work? [16:01:36] (03CR) 10Jforrester: Migrate mobileapps to the service pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [16:12:13] Yippee, build fixed! [16:12:14] Project mediawiki-core-doxygen-docker build #7587: 09FIXED in 8 min 5 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/7587/ [16:16:00] 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 - https://phabricator.wikimedia.org/T225628 (10Legoktm) >>! In T225628#5253647, @Jdforrester-WMF wrote: >>>! In T225628#5253393, @Legoktm wrote: >> I think we can handle that, especially in the new docker based setup. > > Wh... [16:18:29] (03PS2) 10Jforrester: layout: [mediawiki/services/mobileapps] Drop npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [16:18:31] (03PS1) 10Jforrester: layout: [mediawiki/services/mobileapps] Add service-pipeline-test?(-and-publish) alongside [integration/config] - 10https://gerrit.wikimedia.org/r/516650 [16:18:33] (03PS1) 10Jforrester: [DNM] layout: [mediawiki/services/mobileapps/deploy] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/516651 [16:23:38] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/services/mobileapps] Add service-pipeline-test?(-and-publish) alongside [integration/config] - 10https://gerrit.wikimedia.org/r/516650 (owner: 10Jforrester) [16:25:08] (03Merged) 10jenkins-bot: layout: [mediawiki/services/mobileapps] Add service-pipeline-test?(-and-publish) alongside [integration/config] - 10https://gerrit.wikimedia.org/r/516650 (owner: 10Jforrester) [16:27:46] !log Reloading Zuul to add service-pipeline-test?(-and-publish) to mediawiki/services/mobileapps [16:27:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:24] !log Fix Jenkins Console section "setup quibble mw-install" pattern to not slurp the whole output on 'mediawiki-quibble-composertest' jobs [16:28:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:51] Krinkle: Oh, nice, thanks. [16:35:13] (03CR) 10Jforrester: [C: 04-1] "OK, pipeline test is installed for this repo and working. Success!" [integration/config] - 10https://gerrit.wikimedia.org/r/516517 (owner: 10Mholloway) [16:42:45] (03PS1) 10Jforrester: layout: Mark four services' old deploy repos as archived [integration/config] - 10https://gerrit.wikimedia.org/r/516658 [16:58:28] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10mmodell) == [[ https://pywebview.flowrl.com | pywebview ]] - kinda like electron but much simpler and built on python. >pywebview is a light... [17:12:44] (03CR) 10Brian Wolff: "ping" [integration/config] - 10https://gerrit.wikimedia.org/r/512344 (owner: 10Robert Vogel) [17:16:11] (03CR) 10Brian Wolff: [C: 03+1] Remove non-PHP-7.2 tests for "Extension:WebAuthn" [integration/config] - 10https://gerrit.wikimedia.org/r/512344 (owner: 10Robert Vogel) [17:42:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) Things still using node6 as of 2019-06-12: * Services – T210704 **... [17:56:37] (03PS4) 10Jforrester: jjb, layout: [OOUI] Create ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 [18:15:45] (03CR) 10Jforrester: [C: 03+2] jjb, layout: [OOUI] Create ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [18:17:58] (03CR) 10jerkins-bot: [V: 04-1] jjb, layout: [OOUI] Create ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [18:23:50] (03PS5) 10Jforrester: layout: [OOUI] Use ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 [18:23:52] (03PS1) 10Jforrester: jjb: [OOUI] Provide ooui-npm-special-node10-plus-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516664 [18:24:08] (03CR) 10Jforrester: [C: 03+2] jjb: [OOUI] Provide ooui-npm-special-node10-plus-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516664 (owner: 10Jforrester) [18:27:15] (03Merged) 10jenkins-bot: jjb: [OOUI] Provide ooui-npm-special-node10-plus-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516664 (owner: 10Jforrester) [18:32:24] (03CR) 10Jforrester: [C: 03+2] layout: [OOUI] Use ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [18:32:42] !log Deployed new jjb job ooui-npm-special-node10-plus-composer-docker [18:32:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:33:54] (03Merged) 10jenkins-bot: layout: [OOUI] Use ooui-npm-special-node10-plus-composer-docker, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/514164 (owner: 10Jforrester) [18:35:00] !log Reloaded Zuul to use node10 things for OOUI experimentally [18:35:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:36:55] (03PS1) 10Umherirrender: phan: [EventBus] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/516665 [20:09:09] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 27077 bytes in 1.032 second response time [20:09:20] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:10:10] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26482 bytes in 1.009 second response time [20:10:39] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26481 bytes in 0.769 second response time [20:11:20] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 23317 bytes in 0.949 second response time [20:12:49] * ottomata is looking for a someone with super git fu [20:13:09] my question is around git index/stage [20:13:11] and locking [20:13:21] and gitattribute filter [20:13:42] you may be somewhere past my git fu. [20:15:58] i'm trying to get the gitattribute filter clean process to add a new file. [20:16:17] but it seems i can't because the process runs after a stage (e.g. git add), and the repo is locked [20:20:35] you are somewhere past my git fu. :) [20:20:42] :) [20:22:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hashar) Most probably all of those repositories can just be upgraded from Node 6 to... [20:23:35] (03PS1) 10Thcipriani: Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) [20:28:58] I guess you could probably pass the file you're trying to clean to stdin of whatever the clean filter is...although if the repo is locked you can't add the resulting git object to the repo... [20:29:20] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [20:48:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) >>! In T211784#5254546, @hashar wrote: > Most probably all of those... [20:49:18] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [21:01:11] 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 - https://phabricator.wikimedia.org/T225628 (10hashar) The purpose is two folds: * save us from having to maintain some old/obsolete versions of php. Though the Debian maintainer made PHP versions co installable and easier to... [21:01:41] (03PS1) 10Jforrester: layout: [analytics/aqs] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516674 (https://phabricator.wikimedia.org/T211784) [21:01:43] (03PS1) 10Jforrester: layout: [analytics/dashiki] Experiment with generic-node10-docker; no current CI [integration/config] - 10https://gerrit.wikimedia.org/r/516675 (https://phabricator.wikimedia.org/T211784) [21:01:45] (03PS1) 10Jforrester: layout: [analytics/{mediawiki-storage,wikistats2}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516676 (https://phabricator.wikimedia.org/T211784) [21:04:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hashar) For `jsduck`, if a repository is still using it, just break it. It has been... [21:04:11] (03CR) 10Jforrester: [C: 03+2] layout: [analytics/aqs] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516674 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:04:14] (03CR) 10Jforrester: [C: 03+2] layout: [analytics/dashiki] Experiment with generic-node10-docker; no current CI [integration/config] - 10https://gerrit.wikimedia.org/r/516675 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:04:17] (03CR) 10Jforrester: [C: 03+2] layout: [analytics/{mediawiki-storage,wikistats2}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516676 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:04:54] (03CR) 10Hashar: [C: 03+1] layout: [analytics/aqs] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516674 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:05:08] (03CR) 10Hashar: [C: 03+1] layout: [analytics/dashiki] Experiment with generic-node10-docker; no current CI [integration/config] - 10https://gerrit.wikimedia.org/r/516675 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:05:29] (03CR) 10Hashar: [C: 03+1] layout: [analytics/{mediawiki-storage,wikistats2}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516676 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:05:43] (03Merged) 10jenkins-bot: layout: [analytics/aqs] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516674 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:06:21] (03Merged) 10jenkins-bot: layout: [analytics/dashiki] Experiment with generic-node10-docker; no current CI [integration/config] - 10https://gerrit.wikimedia.org/r/516675 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:06:24] (03Merged) 10jenkins-bot: layout: [analytics/{mediawiki-storage,wikistats2}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516676 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:07:34] !log Updating Zuul with https://gerrit.wikimedia.org/r/516674 https://gerrit.wikimedia.org/r/516675 https://gerrit.wikimedia.org/r/516676 [21:07:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:25:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10hashar) >>! In T223602#5245075, @Ladsgroup wrote: >>>! In T223602#5235411, @ArielGlenn wrote: >> @L... [21:43:12] I saw that _(dumps/dcat experimental check) [21:44:42] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) es6 patch was rolled back because at the time I think we were still on es5 and my code wasn't as forward/backward compatible as I t... [21:44:57] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) p:05High→03Normal [21:45:12] apergos: Patch forthcoming. [21:45:19] (03PS1) 10Jforrester: layout: [data-values/value-view] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516682 (https://phabricator.wikimedia.org/T211784) [21:45:21] (03PS1) 10Jforrester: layout: [design/landing-page] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516683 (https://phabricator.wikimedia.org/T211784) [21:45:23] (03PS1) 10Jforrester: layout: [labs/tools/graphql] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516684 (https://phabricator.wikimedia.org/T211784) [21:45:25] (03PS1) 10Jforrester: layout: [mapdata] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516685 (https://phabricator.wikimedia.org/T211784) [21:45:27] (03PS1) 10Jforrester: layout: [operations/dumps/dcat] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516686 (https://phabricator.wikimedia.org/T211784) [21:45:29] (03PS1) 10Jforrester: layout: [react.i18n] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516687 (https://phabricator.wikimedia.org/T211784) [21:45:31] (03PS1) 10Jforrester: layout: [wikidata/build-resources] Archive, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516688 (https://phabricator.wikimedia.org/T211784) [21:45:33] (03PS1) 10Jforrester: layout: [wikimedia-cz/tracker] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516689 (https://phabricator.wikimedia.org/T211784) [21:45:35] (03PS1) 10Jforrester: layout: [wikimedia/fundraising/dash] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516690 (https://phabricator.wikimedia.org/T211784) [21:45:37] (03PS1) 10Jforrester: layout: [wikimedia/portals{,/deploy}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516691 (https://phabricator.wikimedia.org/T211784) [21:45:39] (03PS1) 10Jforrester: layout: [wikimedia/wikimania-scholarships] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516692 (https://phabricator.wikimedia.org/T211784) [21:45:44] *patches [21:46:12] fine by me [21:46:48] Only one patch for _dcat. :-) [21:46:59] (03CR) 10Jforrester: [C: 03+2] layout: [data-values/value-view] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516682 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:48:08] (03CR) 10Jforrester: [C: 03+2] layout: [design/landing-page] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516683 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:48:31] (03CR) 10Jforrester: [C: 03+2] layout: [labs/tools/graphql] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516684 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:48:35] (03Merged) 10jenkins-bot: layout: [data-values/value-view] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516682 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:48:37] (03CR) 10Jforrester: [C: 03+2] layout: [mapdata] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516685 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:49:44] (03CR) 10Jforrester: [C: 03+2] layout: [operations/dumps/dcat] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516686 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:49:47] (03Merged) 10jenkins-bot: layout: [design/landing-page] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516683 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:49:59] (03CR) 10Jforrester: [C: 03+2] layout: [react.i18n] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516687 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:02] (03CR) 10Jforrester: [C: 03+2] layout: [wikidata/build-resources] Archive, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516688 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:11] (03Merged) 10jenkins-bot: layout: [labs/tools/graphql] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516684 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:19] (03Merged) 10jenkins-bot: layout: [mapdata] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516685 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:36] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia-cz/tracker] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516689 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:44] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/fundraising/dash] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516690 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:48] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/portals{,/deploy}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516691 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:50:53] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/wikimania-scholarships] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516692 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:51:22] (03CR) 10Jforrester: [C: 03+2] layout: Mark four services' old deploy repos as archived [integration/config] - 10https://gerrit.wikimedia.org/r/516658 (owner: 10Jforrester) [21:51:56] (03Merged) 10jenkins-bot: layout: [operations/dumps/dcat] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516686 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:51:59] (03Merged) 10jenkins-bot: layout: [react.i18n] Migrate to node10-docker, already passes [integration/config] - 10https://gerrit.wikimedia.org/r/516687 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:02] (03Merged) 10jenkins-bot: layout: [wikidata/build-resources] Archive, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516688 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:17] (03Merged) 10jenkins-bot: layout: [wikimedia-cz/tracker] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516689 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:25] (03Merged) 10jenkins-bot: layout: [wikimedia/fundraising/dash] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516690 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:27] (03Merged) 10jenkins-bot: layout: [wikimedia/portals{,/deploy}] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516691 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:30] (03Merged) 10jenkins-bot: layout: [wikimedia/wikimania-scholarships] Experiment with generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516692 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [21:52:35] #spam [21:52:39] :D [21:52:58] (03Merged) 10jenkins-bot: layout: Mark four services' old deploy repos as archived [integration/config] - 10https://gerrit.wikimedia.org/r/516658 (owner: 10Jforrester) [21:53:05] They're individual patches so they can be easily reverted if we need to. [21:54:01] !log Reloading Zuul to deploy, umm, 12 patches. 🤷🏽‍♂️ [21:54:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:06:28] 10Release-Engineering-Team, 10Operations: GPG Key expired apt repository - https://phabricator.wikimedia.org/T225677 (10Reedy) [22:06:41] 10Release-Engineering-Team, 10Operations: GPG Key expired apt repository - https://phabricator.wikimedia.org/T225677 (10Aklapper) [22:06:45] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Operations, 10Parsoid: signatures were invalid: EXPKEYSIG 90E9F83F22250DD7 MediaWiki releases repository - https://phabricator.wikimedia.org/T225601 (10Aklapper) [22:07:22] PROBLEM - Puppet errors on integration-trigger-01 is CRITICAL: CRITICAL: 2.25% of data above the critical threshold [3.0] [22:11:00] PROBLEM - Free space - all mounts on integration-trigger-01 is CRITICAL: CRITICAL: integration.integration-trigger-01.diskspace._srv.byte_percentfree (<60.00%) [22:11:21] Huh, why would -trigger get disk space used up so quickly? [22:11:28] It's new yesterday, right? [22:12:05] created too small? [22:12:36] CC brennen. [22:16:23] James_F: i think i blew it up trying to apply role::ci::slave::labs::docker [22:16:30] ...which i think i probably don't need [22:18:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [22:18:25] (03PS1) 10Jforrester: layout: [mapdata] Migrate to generic-node10-browser-docker, passes [integration/config] - 10https://gerrit.wikimedia.org/r/516697 [22:18:27] (03PS1) 10Jforrester: jjb: Drop mapdata-npm-node-6-docker and marvin-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516698 [22:18:43] brennen: Possibly. I stay far away from puppet land. [22:18:58] (03CR) 10Jforrester: [C: 03+2] layout: [mapdata] Migrate to generic-node10-browser-docker, passes [integration/config] - 10https://gerrit.wikimedia.org/r/516697 (owner: 10Jforrester) [22:19:02] (03CR) 10Jforrester: [C: 03+2] jjb: Drop mapdata-npm-node-6-docker and marvin-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516698 (owner: 10Jforrester) [22:21:03] (03Merged) 10jenkins-bot: layout: [mapdata] Migrate to generic-node10-browser-docker, passes [integration/config] - 10https://gerrit.wikimedia.org/r/516697 (owner: 10Jforrester) [22:21:17] (03Merged) 10jenkins-bot: jjb: Drop mapdata-npm-node-6-docker and marvin-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516698 (owner: 10Jforrester) [22:21:38] !log Reloading Zuul to make mpadata use generic-node10-browser-docker [22:21:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:22:21] !log Dropped mapdata-npm-node-6-docker and marvin-npm-node-6-docker from jjb [22:22:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:24:42] (03PS1) 10Jforrester: layout: [wikimedia-cz/tracker] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516699 [22:24:44] (03PS1) 10Jforrester: jjb: Drop wikimedia-cz-tracker-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516700 [22:26:05] (03PS1) 10Jforrester: layout: [wikimedia/fundraising/dash] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516701 [22:26:07] (03PS1) 10Jforrester: jjb: Drop fundraising-dash-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516702 [22:28:01] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10mmodell) >>! In T216605#5251322, @thcipriani wrote: > If so, we can apply the generated patch, reindex user accounts and resolve this ticket -- trying to think of any other i... [22:31:38] (03PS1) 10Jforrester: layout: [wikimedia/wikimania-scholarships] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516704 [22:31:40] (03PS1) 10Jforrester: layout: [wikimedia/portals{,/deploy}] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516705 [22:31:42] (03PS1) 10Jforrester: jjb: Drop wikimedia-portals-npm-browser-node-6-docker and wikimedia-portals-deploy-npm-browser-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516706 [22:31:54] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia-cz/tracker] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516699 (owner: 10Jforrester) [22:31:57] (03CR) 10Jforrester: [C: 03+2] jjb: Drop wikimedia-cz-tracker-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516700 (owner: 10Jforrester) [22:32:00] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/fundraising/dash] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516701 (owner: 10Jforrester) [22:32:02] (03CR) 10Jforrester: [C: 03+2] jjb: Drop fundraising-dash-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516702 (owner: 10Jforrester) [22:32:09] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/wikimania-scholarships] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516704 (owner: 10Jforrester) [22:32:15] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/portals{,/deploy}] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516705 (owner: 10Jforrester) [22:32:17] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10Paladox) I concur with ^^. [22:32:19] (03CR) 10Jforrester: [C: 03+2] jjb: Drop wikimedia-portals-npm-browser-node-6-docker and wikimedia-portals-deploy-npm-browser-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516706 (owner: 10Jforrester) [22:33:37] (03Merged) 10jenkins-bot: layout: [wikimedia-cz/tracker] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516699 (owner: 10Jforrester) [22:35:10] (03Merged) 10jenkins-bot: jjb: Drop wikimedia-cz-tracker-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516700 (owner: 10Jforrester) [22:35:12] (03Merged) 10jenkins-bot: layout: [wikimedia/fundraising/dash] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516701 (owner: 10Jforrester) [22:35:15] (03Merged) 10jenkins-bot: jjb: Drop fundraising-dash-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516702 (owner: 10Jforrester) [22:35:17] (03Merged) 10jenkins-bot: layout: [wikimedia/wikimania-scholarships] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516704 (owner: 10Jforrester) [22:35:19] (03Merged) 10jenkins-bot: layout: [wikimedia/portals{,/deploy}] Migrate to generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516705 (owner: 10Jforrester) [22:35:50] (03Merged) 10jenkins-bot: jjb: Drop wikimedia-portals-npm-browser-node-6-docker and wikimedia-portals-deploy-npm-browser-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516706 (owner: 10Jforrester) [22:36:47] (03PS1) 10Jforrester: layout: [3d2png, mediawiki/services/{chromium-render,recommendation-api,trending-edits}] Add service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/516707 (https://phabricator.wikimedia.org/T211784) [22:37:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [22:38:11] !log deleting and re-creating integration-trigger-01 [22:38:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:40:00] !log Dropped jjb jobs wikimedia-cz-tracker-npm-node-6-docker fundraising-dash-npm-node-6-docker wikimedia-portals-npm-browser-node-6-docker wikimedia-portals-deploy-npm-browser-node-6-docker [22:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:40:57] (03CR) 10Jforrester: [C: 03+2] layout: [3d2png, mediawiki/services/{chromium-render,recommendation-api,trending-edits}] Add service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/516707 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [22:41:31] PROBLEM - Host integration-trigger-01 is DOWN: CRITICAL - Host Unreachable (172.16.5.93) [22:42:20] (03Merged) 10jenkins-bot: layout: [3d2png, mediawiki/services/{chromium-render,recommendation-api,trending-edits}] Add service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/516707 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [22:44:09] !log Enabling service-pipeline-test on four services: 3d2png, chromium-render, recommendation-api, trending-edits [22:44:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:49:35] RECOVERY - Host integration-trigger-01 is UP: PING OK - Packet loss = 0%, RTA = 1.53 ms [22:51:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [23:08:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [23:13:47] (03PS1) 10Jforrester: layout: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now [integration/config] - 10https://gerrit.wikimedia.org/r/516716 (https://phabricator.wikimedia.org/T211784) [23:15:59] (03PS2) 10Jforrester: layout: [mediawiki/services/recommendation-api Drop npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516635 (owner: 10Mholloway) [23:16:10] (03PS3) 10Jforrester: layout: [mediawiki/services/recommendation-api] Drop npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/516635 (owner: 10Mholloway) [23:16:14] (03CR) 10Jforrester: [C: 03+2] layout: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now [integration/config] - 10https://gerrit.wikimedia.org/r/516716 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [23:17:45] (03Merged) 10jenkins-bot: layout: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now [integration/config] - 10https://gerrit.wikimedia.org/r/516716 (https://phabricator.wikimedia.org/T211784) (owner: 10Jforrester) [23:18:42] (03PS1) 10Jforrester: layout: [mediawiki/services/recommendation-api/deploy] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/516718 [23:18:44] (03PS1) 10Jforrester: jjb: Drop recommendation-api-npm-node-6-docker and recommendation-api-deploy-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/516719 [23:19:22] (03CR) 10Jforrester: [C: 04-1] "Waiting on production switch-over, I believe?" [integration/config] - 10https://gerrit.wikimedia.org/r/516635 (owner: 10Mholloway) [23:20:25] !log Zuul: Disable service-pipeline-test from 3d2png, chromium-render, and trending-edits for now [23:20:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:20:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) OK, that's enough for today. [23:27:01] PROBLEM - Free space - all mounts on integration-trigger-01 is CRITICAL: CRITICAL: integration.integration-trigger-01.diskspace._srv.byte_percentfree (<12.50%) [23:34:00] !log deleting and rebuilding integration-trigger-01 (again) to use jessie instead of stretch [23:34:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:36:33] PROBLEM - Host integration-trigger-01 is DOWN: CRITICAL - Host Unreachable (172.16.6.54) [23:51:32] RECOVERY - Host integration-trigger-01 is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms [23:55:03] Um the phab maint account just flooded my mail box [23:55:09] Is that expected? [23:55:15] paladox: Sadly, yes. [23:55:19] sorry! [23:55:38] I thought it wouldn't [23:55:47] I'll use the option silencing option for future edits [23:55:49] :(