[00:13:57] (03PS4) 10Bmansurov: Add npm job for recommendation-api service [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) [00:14:22] (03CR) 10jerkins-bot: [V: 04-1] Add npm job for recommendation-api service [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [00:22:06] (03CR) 10Thcipriani: Add npm job for recommendation-api service (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [00:56:44] I think i have some ideas on how to do this now (blacklist) [01:49:18] (03PS7) 10Krinkle: Add mediawiki-fresnel-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/459268 (https://phabricator.wikimedia.org/T133646) [03:01:53] https://gerrit-review.googlesource.com/c/plugins/reviewers-by-blame/+/210812/ [03:01:58] my patch that adds support for it [03:43:39] 10Gerrit, 10Upstream: Exempt libraryupgrader from Gerrit's reviewers-by-blame plugin - https://phabricator.wikimedia.org/T213997 (10Paladox) a:03Paladox https://gerrit-review.googlesource.com/c/plugins/reviewers-by-blame/+/210812/13 [08:53:15] 10Release-Engineering-Team, 10Performance-Team: Create gerrit bot for git pushes to specific repo from prod machines - https://phabricator.wikimedia.org/T214015 (10Gilles) I've figured out how to do that locally with the pygerrit2 python library and it's fairly straightforward. Unfortunately that library [[ ht... [09:06:12] 10Release-Engineering-Team, 10Performance-Team: Create gerrit bot for git pushes to specific repo from prod machines - https://phabricator.wikimedia.org/T214015 (10Gilles) Here's how simple the code is for publishing the change to gerrit with that library: ` auth = HTTPBasicAuth(username, password) gerrit_api... [09:06:55] 10Release-Engineering-Team, 10Performance-Team: Create gerrit bot for git pushes to specific repo from prod machines - https://phabricator.wikimedia.org/T214015 (10Gilles) a:03Gilles [09:08:45] 10Release-Engineering-Team, 10Performance-Team: Backport pygerrit2 to Debian Stretch - https://phabricator.wikimedia.org/T214149 (10Gilles) p:05Triage→03Normal [09:21:20] 10Release-Engineering-Team, 10Operations, 10Performance-Team: Backport pygerrit2 to Debian Stretch - https://phabricator.wikimedia.org/T214149 (10Joe) a:05Gilles→03Joe [09:25:19] 10Continuous-Integration-Config: `Task "karma:main" failed` when running 0 tests - https://phabricator.wikimedia.org/T214140 (10Peachey88) [09:31:04] 10Release-Engineering-Team, 10Performance-Team: Create gerrit bot for git pushes to specific repo from prod machines - https://phabricator.wikimedia.org/T214015 (10Joe) [09:31:09] 10Release-Engineering-Team, 10Operations, 10Performance-Team: Backport pygerrit2 to Debian Stretch - https://phabricator.wikimedia.org/T214149 (10Joe) 05Open→03Resolved [09:42:20] 10Diffusion, 10Wikidata: Update the "Observe" URI of diffusion mirror wikibase-termbox - https://phabricator.wikimedia.org/T214150 (10WMDE-leszek) [09:55:34] 10Diffusion, 10Wikidata: Change the "Observe" URI of diffusion mirror wikibase-termbox - https://phabricator.wikimedia.org/T214150 (10WMDE-leszek) [10:16:39] 10Diffusion, 10Wikidata, 10User-MarcoAurelio: Change the "Observe" URI of diffusion mirror wikibase-termbox - https://phabricator.wikimedia.org/T214150 (10MarcoAurelio) 05Open→03Resolved a:03MarcoAurelio Done. [10:40:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Newsletter, 10Jenkins, 10User-zeljkofilipin: selenium-daily-beta-Newsletter failing during `npm install` - https://phabricator.wikimedia.org/T213944 (10zeljkofilipin) ` integration/config/dockerfiles$ git... [10:42:56] 10Continuous-Integration-Infrastructure, 10docker-pkg: Pruning docker-pkg images - https://phabricator.wikimedia.org/T207703 (10Joe) >>! In T207703#4687233, @Legoktm wrote: > Definitely. I'd also recommend keeping the previous version as well just in case we have to revert. Well the old version will still be... [11:52:11] 10Release-Engineering-Team, 10Performance-Team: Create gerrit bot for git pushes to specific repo from prod machines - https://phabricator.wikimedia.org/T214015 (10Gilles) 05Open→03Resolved Alright, I think that answers it, it's more straightforward than I anticipated. The bot needs I have are actually muc... [12:09:28] 10Release-Engineering-Team (Kanban): Experiment with continuous deployment using Blubberoid - https://phabricator.wikimedia.org/T214158 (10LarsWirzenius) [12:36:02] 10Gerrit, 10Upstream: Exempt libraryupgrader from Gerrit's reviewers-by-blame plugin - https://phabricator.wikimedia.org/T213997 (10MarcoAurelio) Perhaps use `plugin.reviewers-by-blame.ignoreSubjectRegEx` to exclude `^build\:.*` commits? [12:41:14] (03PS1) 10MarcoAurelio: reviewers-by-blame: ignore build upgrades [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485180 (https://phabricator.wikimedia.org/T213997) [12:42:31] (03PS2) 10MarcoAurelio: reviewers-by-blame: ignore build upgrades [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485180 (https://phabricator.wikimedia.org/T213997) [12:45:54] (03Abandoned) 10MarcoAurelio: reviewers-by-blame: ignore build upgrades [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485180 (https://phabricator.wikimedia.org/T213997) (owner: 10MarcoAurelio) [12:47:25] 10Gerrit, 10Patch-For-Review, 10Upstream: Exempt libraryupgrader from Gerrit's reviewers-by-blame plugin - https://phabricator.wikimedia.org/T213997 (10MarcoAurelio) While build: upgrades probably shouldn't trigger adding reviewers in mass, the patch above does not achieve excluding libraryupgrader from bein... [13:04:22] 10Release-Engineering-Team (Kanban): Experiment with continuous deployment using Blubberoid - https://phabricator.wikimedia.org/T214158 (10LarsWirzenius) I think we should start by looking at the steps from a developer pushing a new change to Gerrit. This should trigger Zuul to run a Jenkins job that builds Blu... [13:21:31] 10Release-Engineering-Team (Kanban), 10docker-pkg, 10Patch-For-Review: Option to allow Docker cache when building images - https://phabricator.wikimedia.org/T210438 (10hashar) 05Open→03Resolved Patch completed and merged in. [13:22:43] (03PS1) 10Hashar: Edit Project Config [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 [13:36:07] (03PS2) 10Hashar: reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) [13:36:35] (03PS3) 10Hashar: reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) [13:39:07] (03PS4) 10Hashar: reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) [13:39:12] (03PS5) 10Hashar: reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) [13:39:44] (03PS6) 10Hashar: reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) [13:42:02] (03CR) 10Thcipriani: [C: 03+1] reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) (owner: 10Hashar) [13:44:43] (03CR) 10Thcipriani: [C: 03+1] "I think this feature needs a per-user per-repo/file opt out before it can be used effectively." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) (owner: 10Hashar) [13:46:44] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] reviewers-by-blame: default to not adding any reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/485184 (https://phabricator.wikimedia.org/T101131) (owner: 10Hashar) [13:47:15] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) As a follow up (also posted as a comment to J139): > The Gerrit plugin blindly adds reviewers even when in a few ca... [13:48:35] 10Gerrit, 10Patch-For-Review, 10Upstream: Exempt libraryupgrader from Gerrit's reviewers-by-blame plugin - https://phabricator.wikimedia.org/T213997 (10hashar) I have made the plugin to no more add any reviewers by default (done at `All-Projects.git` level). So you would no more get spam notifications. Thi... [13:50:08] 10Gerrit, 10Patch-For-Review: reviewers-by-blame plugin: only recent reviewers should be pinged - https://phabricator.wikimedia.org/T214097 (10hashar) As a past author of a code, there are good chance that you are qualified to be a reviewers. Then I can understand then changing to a new team or shifting intere... [14:06:31] (03PS5) 10Bmansurov: Add npm job for recommendation-api service [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) [14:07:28] (03CR) 10jerkins-bot: [V: 04-1] Add npm job for recommendation-api service [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [14:12:30] (03CR) 10Bmansurov: "@Thcipriani, I'm not sure why tests are failing. Can you help fix the issue? Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [14:24:40] hashar: I think you have a typo in your second regex on the git reviewer bot page ('module/...' instead of 'modules') [14:24:47] apropos of the email I am about to send [14:55:05] 10Release-Engineering-Team (Backlog), 10Release Pipeline, 10Scoring-platform-team, 10Wikibase-Containers, and 3 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10Ladsgroup) >>! In T210260#4823617, @hashar wrote: > @Ladsgroup if you retry blubber for wik... [15:02:34] thcipriani apparently the bug that affects 2.16 affects 2.15 (project listing, though not as serious) [15:03:19] oh [15:03:21] maybe it is [15:03:23] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/210797 [15:10:57] thcipriani hashar and greg-g i've implemented the blacklist here https://gerrit-review.googlesource.com/c/plugins/reviewers-by-blame/+/210812 [15:12:25] And i will work on supporting a service user (or a user that will add the reviewers instead of it saying the author did it) [15:17:34] paladox: glad to see the your work on the plugin and that's it's getting some good review :) rather than a service user, I thought this was a particularly thoughtful request: https://lists.wikimedia.org/pipermail/wikitech-l/2019-January/091368.html [15:17:53] ah ok [15:18:01] i'll see what i can do with that [15:18:32] thanks for all your work :) [15:19:36] your welcome :) [15:19:49] (btw i worked with david p on that commit)) [15:24:45] i filed https://bugs.chromium.org/p/gerrit/issues/detail?id=10337 thcipriani [15:30:22] thanks, commented [15:34:27] :) [15:58:32] (03PS1) 10Hashar: docker: add missing control files [integration/config] - 10https://gerrit.wikimedia.org/r/485199 [16:07:35] paladox: would our patch handle usernames containing spaces? ;D [16:07:52] I think so [16:07:59] though would need to test [16:07:59] :) [16:28:06] Should i be able to reproduce test failures in wmf-quibble-vendor-mysql-hhvm-docker using the quibble-jessie-hhvm container? Or which is it [16:28:50] i thought it would be quibble-stretch-hhvm (prod app servers run stretch) but that doesn't seem to be in the registry [16:29:54] ebernhardson: I think it should say in the test log which image is being used [16:29:58] got a link to the failure? [16:30:15] addshore: https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/14226/console [16:31:39] ebernhardson: docker-registry.wikimedia.org/releng/quibble-jessie-hhvm:0.0.28 [16:32:20] addshore: excelent, thanks! where was it? I dont see that in ctrl-f [16:32:33] you might need to expand all of the sections or view the full raw log [16:32:57] addshore: oh, i didn't realize when i clicked 'Full Log' it was still collapsing things away [16:33:06] yup :P sneaky collapsing [16:33:12] silly me, thought that would be the full log :P [16:49:06] (03CR) 10Hashar: Add npm job for recommendation-api service (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [16:53:14] ebernhardson: there are some very rough doc at https://doc.wikimedia.org/quibble/#setup [16:53:52] ebernhardson: notably your local environment needs some setup to create a cache directory and point to some mirror of git repositories to speed up the initial cloning [16:54:45] ebernhardson: it might be possible to just mount an existing local installation. The soft runs in the container expecting the code to be in /workspace/src BUT by default it clones/checkout mediawiki core and runs composer install / npm install [16:55:06] hashar (was that question for https://gerrit-review.googlesource.com/c/plugins/reviewers-by-blame/+/210812 )? [16:55:13] those steps can be skipped by passing to quibble --skip-zuul and --skip-deps [16:55:16] i'll be testing later today :) [16:55:42] ebernhardson: tldr: I am overdue on writing several step by step tutorials :/ [16:56:17] paladox: yeah that was the question :-] [16:56:26] ah ok [16:56:44] i pressume yes, but in my head i think it requires encoding so User+User [16:56:46] paladox: anyway, we might just drop the plugin entirely. Though it might be useful for some repositories, it seems using Git/Reviewers is better anyway [16:57:36] ok [16:58:38] hashar: yea i found some of your docs and got it running, but! The tests pass on my local quibble container but not in CI :( [17:00:04] eeeeek :((( [17:00:34] might file order related [17:00:38] or a time race condition :/ [17:00:54] orrr [17:00:58] hashar: i'm sure i'll figure out something :) it could also be pebcak related to trying to setup all the repos without waiting 3 hours for quibble to do it [17:01:21] possibly [17:01:26] i suppose i'll start quibble in an empty dir and let it do it's thing while i look at this [17:01:48] I have a mirror locally [17:02:08] so I just quibble --git-cache /srv/git/mirror (which has stuff like /srv/git/mirror/mediawiki/core.git [17:02:16] and quibble would clone by reference [17:02:26] ahh, that sounds quite useful actually [17:02:42] but it has to be mounted with docker run -v /srv/git/mirror:/srv/git:ro ( /srv/git is inside the container, that is where quibble looks by default) [17:03:10] I cant remember whether it expects repos to be bare (eg mediawiki/core.git) or if it also recognizes regular repos (mediawiki/core checkout which has a .git sub dir) [17:03:26] hashar: one other thing, is it possible to only run phpunit, and pass phpunit some parameters? Basically i want it to run one test file, not the 15k test suite :) [17:03:52] yeah [17:03:57] there are stages listed in the help at the end [17:04:06] --run=phpunit [17:04:23] would clone/install dependencies, install mediawiki then only run the phpunit.php command [17:04:39] to run a specific command instead of the build in stages one would use --commands [17:04:58] which is executed in mediawiki installation directory with all backends spinned [17:05:18] hashar: sweet, thanks [17:05:33] I think one can try --commands=/bin/bash [17:05:40] and might end up dropping in a shell [17:05:50] (would surely want to pass docker run : -it ) [17:05:59] and from the prompt you can just php tests/phpunit.php [17:06:16] or try: --commands='php tests/phpunit.php --something' which might work [17:06:27] ebernhardson: I gotta rush out sorry, the coworking place is closing [17:06:40] have a good week-end [17:13:22] ebernhardson: for the phpunit syntax in particular it's `php tests/phpunit.php /path/to/core-or-extension/tests/phpunit/particular-file --filter nameOfClass/nameOfMethod` [17:14:07] ebernhardson: I also found this documentation page helpful around some of the particulars of how the wikimedia containers should be run locally https://www.mediawiki.org/wiki/Continuous_integration/Docker#Debug_a_container_locally [18:35:47] kostajh: you're welcome :D [18:36:30] kostajh: I actually use some of those as my dev environment now (for anything that doesn't need mw to be installed, e.g. eslint, npm install, npm test etc.) [18:36:58] cool [18:37:15] Krinkle: what OS are you running? [18:38:19] huiskat 1.0 :P [18:39:23] kostajh: It's our Debian CI image. [18:39:38] https://www.irccloud.com/pastebin/cKdP7CTE/ [18:40:49] James_F: ah, I mean on the host OS. [18:41:10] Oh, well I'm running it on Mac OS and I think Krinkle is too. [18:41:20] the file sync performance with Docker for Mac is bad (as usual) so I've stopped using the docker stack recently [18:41:37] https://github.com/Krinkle/dotfiles/blob/818d8fbdff1401bcbb54191b7f537396f3ef5b81/hosts/KrinkleMac/bin/newnode-shell [18:41:50] How bad does it need to be to inhibit git operations? [18:42:06] These start instantly for me, and behave just as if it was without a container in terms of performance. [18:42:36] Note though, I only use the Docker shell for insecure stuff like npm, not git or other things. [18:42:40] cool [18:44:26] James_F: is https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/485059/ okay? [18:44:45] Oh, sorry, forgot to C+2. [18:44:48] it's a manual cherry-pick I made via CLI and I'm not sure [18:44:56] also, the oportunity [18:45:02] is it worth to cherry-pick? [18:45:24] Sure, we pride ourselves on language functionality. [18:45:28] if it's all okay, I'd be glad for the +2 indeed [18:45:41] But as it's going into 1.32.1 we should remove it from the 1.33.0-pre release notes. [18:46:06] but there are just one release-notes-1.33 file right? [18:46:39] Yeah, we'll sync it into HISTORY at some point from the REL1_32 tip. [18:47:06] so nothing for me to worry about then it seems [18:47:10] thank you for +2 [18:47:43] I'm also unable to chmod https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/+/484005/ but I think it should be okay [18:47:53] after all it's something to run locally [18:59:55] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Experiment with continuous deployment using Blubberoid - https://phabricator.wikimedia.org/T214158 (10greg) [19:03:32] (03CR) 10Bmansurov: Add npm job for recommendation-api service (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) (owner: 10Bmansurov) [19:04:00] (03PS6) 10Bmansurov: Add npm job for recommendation-api service [integration/config] - 10https://gerrit.wikimedia.org/r/483225 (https://phabricator.wikimedia.org/T213222) [19:33:47] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Experiment with continuous deployment using Blubberoid - https://phabricator.wikimedia.org/T214158 (10Reedy) [19:39:39] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10Tgr) More upstream bugs: * [[https://bugs.chromium.org/p/gerrit/issues/detail?id=10337|#10337 Support a "Suggest Reviewer" b... [19:45:12] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-Addshore: Allow use of phan 0.8.5+ in wikimedia CI - https://phabricator.wikimedia.org/T174339 (10Legoktm) a:03Legoktm It's not pretty, but I figured out a full, non-breaking, upgrade path. [19:45:31] (03PS1) 10Legoktm: mediawiki-phan: Support newer versions of php-ast and phan [integration/config] - 10https://gerrit.wikimedia.org/r/485241 (https://phabricator.wikimedia.org/T174339) [19:46:52] (03CR) 10jerkins-bot: [V: 04-1] mediawiki-phan: Support newer versions of php-ast and phan [integration/config] - 10https://gerrit.wikimedia.org/r/485241 (https://phabricator.wikimedia.org/T174339) (owner: 10Legoktm) [19:51:59] Reedy: why can I not figure out what you changed in that blubberoid task title? [19:52:06] Experiment with continuous deployment using Blubberoid to [19:52:08] Experiment with continuous deployment using Blubberoid. [19:52:11] greg-g: removed a double space [19:52:21] I dunno why, but phab removes them in some displays from what I can see [19:52:23] huh, it is hidden in the log line [19:52:25] weird [19:52:30] But in IRC lines, emails etc, it's clearly there [19:52:32] I thought I was craz`y [19:52:39] -` [19:52:48] "Experiment with continuous deployment using Blubberoid" [19:52:52] "Experiment with continuous deployment using Blubberoid" [19:52:57] ack, thanks :) [20:04:24] (03PS2) 10Legoktm: mediawiki-phan: Support newer versions of php-ast and phan [integration/config] - 10https://gerrit.wikimedia.org/r/485241 (https://phabricator.wikimedia.org/T174339) [20:26:48] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-Addshore: Allow use of phan 0.8.5+ in wikimedia CI - https://phabricator.wikimedia.org/T174339 (10Legoktm) Here's the summary on how to upgrade: * Set .extra.phan in composer.json for the specific phan version (just like phan-taint-check-plugin),... [20:29:44] (03Abandoned) 10Legoktm: Use mediawiki-phan:0.1.5 for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/460203 (owner: 10Legoktm) [20:29:49] (03Abandoned) 10Legoktm: Use mediawiki-phan:0.1.5 for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/460204 (owner: 10Legoktm) [20:34:27] (03CR) 10Legoktm: "* run-core.sh still needs updating" [integration/config] - 10https://gerrit.wikimedia.org/r/485241 (https://phabricator.wikimedia.org/T174339) (owner: 10Legoktm) [21:44:59] legoktm: maybe I should be more bold, but been holding off on merging https://gerrit.wikimedia.org/r/#/c/integration/config/+/482527/ because it's a common base image. [21:45:11] maybe you can give it sanity look over based on past experience? [21:45:48] * legoktm looks [21:48:46] Krinkle: does ENV FOO=$BAR work? https://docs.docker.com/engine/reference/builder/#env doesn't make it obvious that it does [21:49:00] I guess we're already using it [21:49:14] in BABEL_CACHE_PATH [21:49:55] (03CR) 10Legoktm: [C: 03+1] "LGTM." [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [21:50:00] I'll check locally env just in case. [21:50:26] (03PS1) 10Thcipriani: Scap: Add CI for move back to gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/485331 [21:51:30] welcome back scap :) [21:51:36] legoktm: I guess I need to bump node10-test-browser as well, right? [21:51:49] yeah [21:52:38] legoktm: :) [21:53:18] (03PS8) 10Krinkle: Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) [21:54:53] (03PS9) 10Krinkle: Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) [22:02:42] gitiles is becomming a core gerrit plugin now! [22:16:02] 10Release-Engineering-Team (Backlog), 10Release Pipeline, 10Scoring-platform-team, 10Wikibase-Containers, and 3 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10greg) 05Open→03Resolved >>! In T210260#4892443, @Ladsgroup wrote: >>>! In T210260#48236... [22:22:47] legoktm: yeah, works as expected [22:25:06] (03PS10) 10Krinkle: Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) [22:25:18] (03CR) 10Krinkle: [C: 03+2] Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:26:24] (03CR) 10jerkins-bot: [V: 04-1] Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:26:36] (03CR) 10jerkins-bot: [V: 04-1] Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:27:54] (03PS11) 10Krinkle: Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) [22:27:59] tabulations [22:28:04] You don't hear that every day [22:28:12] (03CR) 10Krinkle: [C: 03+2] Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:29:34] (03Merged) 10jenkins-bot: Fix nyc and npm-update bugs due to unwritable HOME [integration/config] - 10https://gerrit.wikimedia.org/r/482527 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:31:38] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/482527 / T212602 [22:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:31:40] T212602: Docker-based npm jobs have unwritable homedir - https://phabricator.wikimedia.org/T212602 [22:32:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Patch-For-Review: Docker-based npm jobs have unwritable homedir - https://phabricator.wikimedia.org/T212602 (10Krinkle) p:05Triage→03Normal a:05hashar→03Krinkle [22:35:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Performance-Team, 10Patch-For-Review: npm warns about unwritable $HOME/.config - https://phabricator.wikimedia.org/T213014 (10Krinkle) p:05Triage→03Normal a:03Krinkle [22:35:49] (03PS1) 10Krinkle: Update node10 jobs to latest dockerfile version [integration/config] - 10https://gerrit.wikimedia.org/r/485336 (https://phabricator.wikimedia.org/T212602) [22:38:36] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2510 MB (5% inode=62%) [22:39:52] !log contint1001 - apt-get clean - disk space low [22:39:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:42:21] mutante: thanks, I guess my docker-pgk rebuilding is contributing somehow [22:42:39] it's rebuilding a lot of them, due to a change in the ci-stretch base image. [22:42:58] actually, no it isn't. Nevermind. [22:44:20] (03CR) 10Krinkle: [C: 03+2] "Recompiled and uploaded the 10 affected jobs, per integration-config-jjb-diff-docker output." [integration/config] - 10https://gerrit.wikimedia.org/r/485336 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:46:59] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Unify configuration for local build-context copies and variant artifacts - https://phabricator.wikimedia.org/T211625 (10dduvall) [22:47:04] (03Merged) 10jenkins-bot: Update node10 jobs to latest dockerfile version [integration/config] - 10https://gerrit.wikimedia.org/r/485336 (https://phabricator.wikimedia.org/T212602) (owner: 10Krinkle) [22:48:27] legoktm: hm.. docker: Error response from daemon: manifest for docker-registry.wikimedia.org/releng/node10-test-browser:0.2.2 not found. [22:48:37] [contint1001.wikimedia.org] out: => Building image docker-registry.discovery.wmnet/releng/node10-test-browser:0.2.2 [22:48:37] [contint1001.wikimedia.org] out: == Step 2: publishing [22:48:50] Would suggest it should be done by now, given I don't need the latest tags for this. [22:49:05] (the error is from a Jenkins job) [22:49:35] Ignore if this doesn't sound familiar - just wondering if I forgot something [22:51:37] Hm.. well, now that all the tags are done, it's working. nvm :) [22:52:31] Krinkle: /var/lib/docker/overlay2 is like 31G of 46 total [22:53:43] mutante: The change was pretty small, so I guess it's just slow build up [22:53:46] * Krinkle tries to find graphs [22:54:00] we should probably have a strategy for cleaning up older version of Ci images that aren't used. [22:54:13] if we don't already have soemthing for that - assuming that's where they are stored, I don't know. [22:55:24] https://grafana.wikimedia.org/d/000000377/host-overview?panelId=12&fullscreen&orgId=1&var-server=contint1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=ci&from=1546904126553&to=1547852116992 [22:57:43] !log contint1001 - moved zuul logs from 2018 and gzipped zuul logs from /var/log/zuul to /srv/logs/zuul to free disk space on / [22:57:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:00:40] RECOVERY - Disk space on contint1001 is OK: DISK OK [23:00:43] !log contint1001 - gzipping more files in /var/log/zuul/ [23:00:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:00:52] Krinkle: yep, slow build up. happened before i think [23:00:54] !log Some docker builds on integration-slave-docker-1021 failing with ENOMEM [23:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:01:01] https://integration.wikimedia.org/ci/job/fresnel-node10-browser-docker/30/console [23:01:08] happened three times in a row. [23:01:16] not at the same point during the test [23:02:54] Krinkle: oh.. already ticket https://phabricator.wikimedia.org/T207707 [23:05:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) This happened today. Was about to make a ticket for it and found this. 17:38 < icinga-wm> PROBLEM -... [23:14:22] (03PS1) 10Dduvall: Unify `copies` and `artifacts` configuration [blubber] - 10https://gerrit.wikimedia.org/r/485340 (https://phabricator.wikimedia.org/T211625) [23:15:04] (03CR) 10jerkins-bot: [V: 04-1] Unify `copies` and `artifacts` configuration [blubber] - 10https://gerrit.wikimedia.org/r/485340 (https://phabricator.wikimedia.org/T211625) (owner: 10Dduvall) [23:15:39] (03PS2) 10Dduvall: Unify `copies` and `artifacts` configuration [blubber] - 10https://gerrit.wikimedia.org/r/485340 (https://phabricator.wikimedia.org/T211625) [23:16:18] (03CR) 10jerkins-bot: [V: 04-1] Unify `copies` and `artifacts` configuration [blubber] - 10https://gerrit.wikimedia.org/r/485340 (https://phabricator.wikimedia.org/T211625) (owner: 10Dduvall) [23:20:28] (03PS3) 10Dduvall: Unify `copies` and `artifacts` configuration [blubber] - 10https://gerrit.wikimedia.org/r/485340 (https://phabricator.wikimedia.org/T211625) [23:22:09] !log contint1001: sudo docker image prune # Total reclaimed space: 3.592GB [23:22:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:22:39] hashar: :) [23:23:26] mutante: :D I still have to clean up the doc, will do that next week [23:23:35] as well as audit whatever is left to be done! [23:23:46] hashar: ok, cool! [23:24:13] enjoy the weekend [23:24:32] marxarelli: thcipriani: there are some services containers pilling up on contint1001 and filling / (known issue we have a task to move Docker images to a different partition) [23:24:52] so I just clean containers/images whenever there is an alarm and that seems sufficient! [23:24:58] mutante: thanks you too! [23:25:45] hashar: we could use icinga event handlers to run prune when it gets to 95% [23:25:46] hashar: oh right. we really need a clean-up job for that [23:25:50] thanks for taking care of it [23:26:08] mutante: oh that's an interesting idea too [23:26:17] as opposed to a periodic jenkins job [23:26:21] mutante: as a last recourse / dead man switch yeah [23:26:37] I would rather fix up the root cause of Docker images being on the / partition image [23:26:57] sure. there is lot more space in /srv [23:27:14] I think I / we did the analysis on some task. I lack confidence to repartition the lvm volume though :D [23:27:19] even if we find/allocate more space, we should still prune periodically [23:27:24] or the periodic job can check how much space is left and run only if needed [23:27:42] the ticket is https://phabricator.wikimedia.org/T207707 [23:27:51] mutante: yup /srv is big, but I would rather split the jenkins artifacts and the docker images on different partitions [23:28:15] so that one does not bring down the other whenever some disk get full of inodes/spaces or whatever ;) [23:28:24] sounds good [23:28:35] but hmm it is too late in the week to think about it right now [23:28:55] and I have to solder a bunch of cables in my car tomorrow :(( [23:36:02] (03CR) 10Hashar: [C: 04-1] Scap: Add CI for move back to gerrit (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/485331 (owner: 10Thcipriani) [23:36:24] damn [23:36:46] I have been off for six hours and already got a hundred of email from gerrit ;D [23:36:55] (luckily the blame plugin has been disabled *grin*) [23:37:03] have a good week-end everyone! [23:37:32] hashar: i'll add you to some more .. :o and good night :) [23:38:13] :D