[00:07:12] RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [02:13:32] paladox: I just came across the Weblink functionality for Gerrit (cf: https://gerrit.googlesource.com/plugins/gitiles/+/stable-2.13/src/main/java/com/googlesource/gerrit/plugins/gitiles/GitilesWeblinks.java?autodive=0%2F%2F). Wouldn't that be easier to do for Phab links instead of the hacky [gitweb] config? [02:13:46] (going afk, just wanted to drop that here for posterity) [02:14:43] Seem far more flexible [02:15:20] And could, in theory, make it easier to share the same mapping config we use for Phab, so less custom routing code needed on that end? [02:15:25] * no_justification is just brain dumping [02:36:47] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<10.00%) [04:22:57] Yippee, build fixed! [04:22:57] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #551: 09FIXED in 26 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/551/ [05:36:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:04:45] (03CR) 10Hashar: [C: 032] Replace cicalese@mitre.org with cindom@gmail.com. [integration/config] - 10https://gerrit.wikimedia.org/r/384900 (owner: 10Cicalese) [06:05:50] (03Merged) 10jenkins-bot: Replace cicalese@mitre.org with cindom@gmail.com. [integration/config] - 10https://gerrit.wikimedia.org/r/384900 (owner: 10Cicalese) [06:36:11] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3695669 (10Joe) I took a look at how puppeteer downloads chromium and it's underwhelmning to be honest: they plainl... [06:39:41] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3695672 (10Paladox) Should the releng tag be added to get there advice? [06:41:19] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3695674 (10Joe) >>! In T177276#3689890, @Addshore wrote: > 1 more thing to throw into the mix. > > Right now we h... [06:43:51] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3695675 (10Joe) @Paladox actually I'd open a new ticket describing the problem instead of requesting a change of co... [06:49:07] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3695676 (10Paladox) Ok. Thanks. I will let that for @bmansurov to do :) [06:57:18] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:01:45] RECOVERY - Free space - all mounts on deployment-eventlog02 is OK: OK: All targets OK [07:11:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:34:29] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:34:49] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [07:39:25] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [07:44:48] RECOVERY - Puppet errors on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:45] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3695736 (10Dzahn) a:03demon [07:49:11] 10Gerrit, 10Release-Engineering-Team, 10DBA, 10Operations, 10Security: Gerrit: Convert gerrit's db caractor encoding from utf8 to utf8mb4 to prevent truncation of astral characters - https://phabricator.wikimedia.org/T153899#3695742 (10Dzahn) [07:49:16] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3695740 (10Dzahn) 05Open>03declined >>! In T145885#3663481, @Marostegui wrote: > Can this be closed then? I... [07:49:24] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:56:20] (03CR) 10Hashar: [C: 032] Add Groovier1 to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/384926 (owner: 10Gergő Tisza) [07:58:11] (03Merged) 10jenkins-bot: Add Groovier1 to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/384926 (owner: 10Gergő Tisza) [07:59:37] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3695772 (10Paladox) But it will be fixed when we migrate to notedb. [08:02:48] (03CR) 10Hashar: WIP Created mediawiki-core-qunit-selenium-jessie template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/384961 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [08:03:18] (03CR) 10Hashar: "This looks like a duplicate of https://gerrit.wikimedia.org/r/#/c/384965/ isn't it?" [integration/config] - 10https://gerrit.wikimedia.org/r/384968 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [08:04:56] (03CR) 10Hashar: WIP Running mediawiki-core-qunit-selenium-jessie for Popups (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [08:05:14] zeljkof: I am all confused by your Zuul layout changes at https://gerrit.wikimedia.org/r/#/q/topic:T177262 :D [08:05:14] T177262: [subtask] Rewrite Page Previews browser tests in Node.js - https://phabricator.wikimedia.org/T177262 [08:05:40] hashar: I was confused too! :) let me see [08:05:43] zeljkof: there is lot of overlaps between them! It seems to me we want a change that introduce the new zuul template [08:05:49] and switch projects to it [08:05:53] hashar: ah, yes [08:05:57] then another change that add it for Popups [08:06:04] and another one that remove the old selenium job [08:06:14] hashar: this is the summary https://phabricator.wikimedia.org/T177262#3693519 [08:06:22] I usually send those changes as a chain :] [08:06:58] hashar: I was not sure how to do it, so I want to add qunit-selenium job to popups [08:07:21] but pupups has some fancy yaml that when touched makes big diffs [08:07:49] probably a combination of those should work [08:07:59] we can pair on it, now or during our 1:1 [08:08:07] well [08:08:19] lets do a change that introduce a new template [08:08:43] that would have the mediawiki-core-qunit-selenium-jessie job in test: gate-and-submit: and gate-and-submit-swat [08:09:23] mediawiki/extensions/RelatedArticles can then get that template applied [08:09:43] and would additionally have the job triggered in gate-and-submit-swat [08:10:10] and you can apply that template to mediawiki/core and mediawiki/vendor as well [08:10:32] and to whatever other repo use it (eg Vector) [08:11:14] ok, sounds good, will do that [08:11:17] then adding the job to Popups would be just applying that template [08:11:58] bonus: since RelatedArticles does not trigger the job in gate-and-submit-swat , applying the template would show a change for that repo. So potentially that can be switched in yet another change [08:12:10] the diff would be nicer as a result :] [08:12:26] I guess you can introduce the new template and have nothing shown in the diff [08:12:34] I'll start with that [08:12:39] good thinking [08:12:44] so yeah [08:13:15] 1) introduce a new zuul template that has test / gate / gate-and-submit AND switch core/vendor/Vector to it # should be almost a noop in zuul diff [08:13:43] 2) switch RelatedArticles to the template. Side effect: the job get triggered in gate-and-submit-swat (that is a good thing) [08:13:48] 3) apply the template to Popups [08:14:12] zeljkof: ^^ the summary. That should be three changes probably to make the zuul-diff easier to review :] [08:14:51] hashar: thanks, will do [08:15:11] in the middle of something else, will switch to that in a few minutes [08:15:21] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:00:33] (03PS5) 10Zfilipin: Created mediawiki-core-qunit-selenium-jessie template [integration/config] - 10https://gerrit.wikimedia.org/r/384961 (https://phabricator.wikimedia.org/T177262) [09:02:03] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 10Pywikibot-tests: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#3696046 (10hashar) The tests are tagged with `familly = wikipedia` b... [09:02:03] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:03:09] 10Continuous-Integration-Config, 10Patch-For-Review, 10WorkType-NewFunctionality: Jobs should checkout source in a subdirectory of the workspace to avoid polluting it with CI related files - https://phabricator.wikimedia.org/T130208#3696057 (10hashar) 05Open>03declined With the migration toward container... [09:03:11] 10Continuous-Integration-Scaling, 10RelEng-Archive-FY201718-Q1, 10Pywikibot-i18n, 10Patch-For-Review, 10WorkType-NewFunctionality: Migrate pywikibot/i18n npm job to Nodepool - https://phabricator.wikimedia.org/T130207#3696059 (10hashar) [09:10:28] (03PS1) 10Hashar: RelatedArticles trigger qunit/Selenium for swat [integration/config] - 10https://gerrit.wikimedia.org/r/385156 [09:11:46] (03PS6) 10Hashar: Created mediawiki-core-qunit-selenium-jessie template [integration/config] - 10https://gerrit.wikimedia.org/r/384961 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [09:17:36] (03CR) 10Hashar: [C: 032] RelatedArticles trigger qunit/Selenium for swat [integration/config] - 10https://gerrit.wikimedia.org/r/385156 (owner: 10Hashar) [09:19:17] (03Merged) 10jenkins-bot: RelatedArticles trigger qunit/Selenium for swat [integration/config] - 10https://gerrit.wikimedia.org/r/385156 (owner: 10Hashar) [09:19:19] (03CR) 10Hashar: [C: 032] "Noop in Zuul (beside the job ordering :D )" [integration/config] - 10https://gerrit.wikimedia.org/r/384961 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [09:20:00] (03Merged) 10jenkins-bot: Created mediawiki-core-qunit-selenium-jessie template [integration/config] - 10https://gerrit.wikimedia.org/r/384961 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [09:41:54] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3696178 (10phuedx) ^ Per T178189#3695008 onwards. [09:59:03] (03PS1) 10Giuseppe Lavagetto: Add docker-pkg automatic testing via tox [integration/config] - 10https://gerrit.wikimedia.org/r/385163 [10:00:54] (03CR) 10Hashar: [C: 032] Add docker-pkg automatic testing via tox [integration/config] - 10https://gerrit.wikimedia.org/r/385163 (owner: 10Giuseppe Lavagetto) [10:02:12] (03Merged) 10jenkins-bot: Add docker-pkg automatic testing via tox [integration/config] - 10https://gerrit.wikimedia.org/r/385163 (owner: 10Giuseppe Lavagetto) [10:13:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:19:06] (03Abandoned) 10Zfilipin: WIP Run mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384968 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [10:19:21] (03PS3) 10Zfilipin: WIP Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) [10:21:02] (03CR) 10jerkins-bot: [V: 04-1] WIP Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [10:28:46] (03PS4) 10Zfilipin: Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) [10:30:33] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:53:21] PROBLEM - SSH on deployment-videoscaler01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:09] RECOVERY - SSH on deployment-videoscaler01 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u1 (protocol 2.0) [10:59:46] 10Gerrit, 10Developer-Relations, 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3696301 (10Aklapper) a:03Aklapper Indeed, https://www.mediawiki.org/wiki/Gerrit/Getting_started looks like a great how-to. One thing left to... [11:01:13] hashar: the popups diff is still huge :( https://gerrit.wikimedia.org/r/#/c/384965/ cc phuedx jdlrobson [11:01:26] I'll try a smaller commit [11:05:24] (03PS1) 10Zfilipin: WIP Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/385172 (https://phabricator.wikimedia.org/T177262) [11:07:41] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3696324 (10zeljkofilipin) Screenshots are there: - https://integration.wikimedia.o... [11:10:31] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:11:47] hashar: the minimal commit still produces a huge diff :( https://gerrit.wikimedia.org/r/#/c/385172/ [11:13:33] 00:00:10.284 -DEBUG:zuul.DependentPipelineManager:Created queue: [11:13:33] 00:00:10.285 +DEBUG:zuul.DependentPipelineManager:Created queue: [11:13:48] zeljkof: yeah it no more creates the mediawiki/extensions/Popups queue [11:13:59] zeljkof: it is just an order issue no worries :) [11:14:24] (03Abandoned) 10Zfilipin: WIP Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/385172 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [11:15:08] (03PS5) 10Zfilipin: Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) [11:15:44] hashar: do you think https://gerrit.wikimedia.org/r/#/c/384965/ could be merged then? [11:16:28] zeljkof: i guess ? Popups was the last to use the extension-selenium-gate template wasn't it ? [11:16:45] hashar: yes, did not find anything else using it [11:16:51] that's why I have delete it [11:17:01] then lets do it :) [11:17:04] (03CR) 10Hashar: [C: 031] Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [11:17:08] ok, merging and deploying [11:17:17] (03CR) 10Zfilipin: [C: 032] Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [11:17:27] hashar: thanks! :) [11:18:12] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.5.1 - https://phabricator.wikimedia.org/T156120#3696344 (10Paladox) [11:19:19] (03Merged) 10jenkins-bot: Running mediawiki-core-qunit-selenium-jessie for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384965 (https://phabricator.wikimedia.org/T177262) (owner: 10Zfilipin) [11:21:00] !log Reloading Zuul to deploy 26f4ff51f9b119f040b32fea4b45910cf14635c4 [11:21:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:23:22] 10Gerrit, 10Developer-Relations, 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3696351 (10Paladox) I think we should include both the manual way and git-review. Since many users are finding using http clones with git revie... [11:34:42] 10Gerrit, 10Developer-Relations, 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3696376 (10Aklapper) What is "the manual way"? Please be clear and provide specific commands to make sure we talk about the same things. How-to... [11:43:26] 10Gerrit, 10Developer-Relations, 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3696411 (10Paladox) Manual command I mean is something like git push, git pull. [12:06:48] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3696438 (10Addshore) Okay! mediawiki-phan-0.8 as an image name and then we can leave the tag for other versioning... [12:06:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3696439 (10hashar) 05Open>03Resolved There were some follow up patches required b... [12:25:59] 10Continuous-Integration-Infrastructure (shipyard): Should we expose some JENKINS_ environment variables in docker? - https://phabricator.wikimedia.org/T177684#3696464 (10hashar) An alternative would be to blindly pass the whole environment to the Docker container? /usr/bin/env > .env [12:37:00] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Find CI container build location - https://phabricator.wikimedia.org/T173128#3696471 (10hashar) That will be on the ci::master hosts. Eg contint1001 / contint2001. [12:44:55] (03PS4) 10Hashar: build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 [12:46:50] 10Release-Engineering-Team, 10Electron-PDFs, 10Proton, 10Readers-Web-Backlog, and 2 others: How should we get Chromium for use in puppeteer? - https://phabricator.wikimedia.org/T178570#3696477 (10bmansurov) [12:47:28] (03CR) 10Hashar: [C: 032] "kwargs -> *kwargs -> **kwargs" [integration/config] - 10https://gerrit.wikimedia.org/r/384553 (owner: 10Hashar) [12:48:13] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10bmansurov) OK, {T178570} has been created. Let's continue discussing the findings from T178189#3695008 i... [12:48:37] (03Merged) 10jenkins-bot: build.py: support dry run [integration/config] - 10https://gerrit.wikimedia.org/r/384553 (owner: 10Hashar) [12:57:56] (03PS1) 10Hashar: docker:ARG DEBIAN_FRONTEND=noninteractive is not herited [integration/config] - 10https://gerrit.wikimedia.org/r/385180 [12:59:50] (03PS2) 10Hashar: docker:ARG DEBIAN_FRONTEND=noninteractive is not herited [integration/config] - 10https://gerrit.wikimedia.org/r/385180 [13:04:28] Yippee, build fixed! [13:04:29] Project selenium-Math » firefox,beta,Linux,BrowserTests build #549: 09FIXED in 27 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/549/ [13:31:34] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:33:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3696585 (10Paladox) Your welcome :) [13:38:07] (03PS3) 10Hashar: docker:ARG DEBIAN_FRONTEND=noninteractive is not herited [integration/config] - 10https://gerrit.wikimedia.org/r/385180 [13:39:06] (03CR) 10Hashar: [C: 032] docker:ARG DEBIAN_FRONTEND=noninteractive is not herited [integration/config] - 10https://gerrit.wikimedia.org/r/385180 (owner: 10Hashar) [13:41:50] (03Merged) 10jenkins-bot: docker:ARG DEBIAN_FRONTEND=noninteractive is not herited [integration/config] - 10https://gerrit.wikimedia.org/r/385180 (owner: 10Hashar) [14:00:14] (03PS1) 10Hashar: docker: set APT::Install-Recommends "false" [integration/config] - 10https://gerrit.wikimedia.org/r/385188 [14:02:22] (03PS5) 10Hashar: build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 [14:04:30] (03CR) 10Hashar: [C: 032] build.py decouple find_tree from gen_deps_tree (037 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/384554 (owner: 10Hashar) [14:04:35] (03PS4) 10Hashar: build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 [14:05:26] (03CR) 10Hashar: [C: 04-1] build.py: refactor dep tree to use 'scratch' [integration/config] - 10https://gerrit.wikimedia.org/r/384580 (owner: 10Hashar) [14:05:38] (03Merged) 10jenkins-bot: build.py decouple find_tree from gen_deps_tree [integration/config] - 10https://gerrit.wikimedia.org/r/384554 (owner: 10Hashar) [14:06:03] (03CR) 10Hashar: [C: 032] build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 (owner: 10Hashar) [14:06:32] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:07:06] (03Merged) 10jenkins-bot: build.py: split gen_deps_tree in two functions [integration/config] - 10https://gerrit.wikimedia.org/r/384573 (owner: 10Hashar) [14:38:16] (03PS4) 10Hashar: Convert tabs to 4 spaces [integration/config] - 10https://gerrit.wikimedia.org/r/381286 [14:45:24] (03CR) 10Hashar: [C: 032] "Bye bye tabulations!" [integration/config] - 10https://gerrit.wikimedia.org/r/381286 (owner: 10Hashar) [14:48:10] (03Merged) 10jenkins-bot: Convert tabs to 4 spaces [integration/config] - 10https://gerrit.wikimedia.org/r/381286 (owner: 10Hashar) [15:10:10] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:16:46] HI [15:17:02] Can you deploy this: https://gerrit.wikimedia.org/r/#/c/385198/ [15:17:16] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10monitoring: Icinga disk space alert when a Docker container is running on an host - https://phabricator.wikimedia.org/T178454#3692868 (10thcipriani) This also depends on the storag... [15:17:31] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10monitoring: Icinga disk space alert when a Docker container is running on an host - https://phabricator.wikimedia.org/T178454#3696811 (10thcipriani) p:05Triage>03Normal [15:19:36] Please, if you can deploy this: https://gerrit.wikimedia.org/r/#/c/385198/ [15:21:51] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:34:03] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:51:49] (03PS1) 10Hashar: Expose JENKINS_URL in Docker [integration/config] - 10https://gerrit.wikimedia.org/r/385204 (https://phabricator.wikimedia.org/T177684) [15:54:25] (03CR) 10Hashar: [C: 032] Expose JENKINS_URL in Docker [integration/config] - 10https://gerrit.wikimedia.org/r/385204 (https://phabricator.wikimedia.org/T177684) (owner: 10Hashar) [15:54:48] (03CR) 10Hashar: [C: 032] "Changed:" [integration/config] - 10https://gerrit.wikimedia.org/r/385204 (https://phabricator.wikimedia.org/T177684) (owner: 10Hashar) [15:55:37] (03Merged) 10jenkins-bot: Expose JENKINS_URL in Docker [integration/config] - 10https://gerrit.wikimedia.org/r/385204 (https://phabricator.wikimedia.org/T177684) (owner: 10Hashar) [16:30:10] (03Abandoned) 10Awight: Tests for DonationInterface/vendor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/304875 (https://phabricator.wikimedia.org/T143025) (owner: 10Awight) [16:40:46] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867#3697107 (10dduvall) [16:40:48] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Include Blubber metadata in Dockerfile output as labels - https://phabricator.wikimedia.org/T178022#3697105 (10dduvall) 05Open>03Resolved [16:45:40] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubber config input validation - https://phabricator.wikimedia.org/T175186#3697114 (10dduvall) p:05Triage>03Normal a:03dduvall [16:52:53] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Release Pipeline: On CI, upgrade docker-ce from 17.06.2 to 17.09.0 - https://phabricator.wikimedia.org/T177499#3697129 (10hashar) [16:53:15] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Release Pipeline: Update docker image docker-registry.wikimedia.org/wikimedia-jessie - https://phabricator.wikimedia.org/T177055#3697130 (10hashar) [16:54:05] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/e6ae8b80f4ba373fd89095d97c351cdb36fafbec11592ca825a68d2a37786786/merged is not accessible: Permission denied [16:54:55] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: build script should build dependant containers in order - https://phabricator.wikimedia.org/T176318#3697132 (10hashar) 05Open>03Resolved a:03hashar This is mostly working. Child images can be rebuild by passing `--recurse`. Ideally... [16:57:53] 10Continuous-Integration-Infrastructure (shipyard): build CI images after a merge on integration/config - https://phabricator.wikimedia.org/T178594#3697153 (10hashar) [17:01:40] (03PS1) 10Hashar: Build docker images on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) [17:02:09] (03CR) 10jerkins-bot: [V: 04-1] Build docker images on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) (owner: 10Hashar) [17:03:01] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: build CI images after a merge on integration/config - https://phabricator.wikimedia.org/T178594#3697175 (10hashar) a:03hashar [17:03:03] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697177 (10demon) I'm not sure why we need another task or to tag RelEng since I've been following along since the... [17:12:05] RECOVERY - Disk space on contint1001 is OK: DISK OK [17:15:05] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/713ae5d770fa652eea12a398a27d6f7fbe926b4da65182b4f02269d4e44f26b6/merged is not accessible: Permission denied [17:22:05] RECOVERY - Disk space on contint1001 is OK: DISK OK [17:29:06] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/b4ca3300c880c818b7096f889e4e14d7ea5c29355bbd767166739299304cf2fb/merged is not accessible: Permission denied [17:29:32] 10Release-Engineering-Team (Watching / External), 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#3697322 (10debt) After we switch to node.js, we won't have this issue any more. We'll check a... [17:29:40] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform - https://phabricator.wikimedia.org/T174103#3550983 (10debt) [17:29:42] 10Release-Engineering-Team (Watching / External), 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#3697326 (10debt) [17:31:06] RECOVERY - Disk space on contint1001 is OK: DISK OK [17:37:52] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697339 (10bmansurov) @demon, the other task is about finding out whether we need to submit the Chromium binary int... [17:53:28] (03PS2) 10Hashar: Build docker images on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) [17:53:34] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697378 (10demon) Well I'm implementing it anyway, this isn't the only use case. [17:57:07] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/0a44b085d526ad80ed40ad34793b3200ae67d5b256ddb502377c455f48e0a784/merged is not accessible: Permission denied [17:59:52] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697386 (10bmansurov) @demon, that's great. Any idea when it'll be ready? [18:00:17] (03PS6) 10Dduvall: WIP Service pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/380551 (https://phabricator.wikimedia.org/T175297) [18:06:38] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697427 (10demon) Hopefully next week, we've already got some patches in flight for this to start configuring it. [18:31:11] bazel-genfiles/plugins/lfs/lfs.jar :). [18:31:42] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697537 (10bmansurov) Awesome! [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:44:26] RECOVERY - Disk space on contint1001 is OK: DISK OK [18:49:24] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/4fed7de4567b78fc7bb502bece45d1794120189a2ab399a875ad5dfab5e40eb8/merged is not accessible: Permission denied [18:53:34] ACKNOWLEDGEMENT - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /var/lib/docker/overlay2/4fed7de4567b78fc7bb502bece45d1794120189a2ab399a875ad5dfab5e40eb8/merged is not accessible: Permission denied daniel_zahn https://phabricator.wikimedia.org/T178454 [18:54:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10netops, 10Jenkins: Disable Jenkins autodiscovery system - https://phabricator.wikimedia.org/T178608#3697616 (10hashar) [18:55:22] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10monitoring: Icinga disk space alert when a Docker container is running on an host - https://phabricator.wikimedia.org/T178454#3692868 (10Dzahn) I disabled Icinga notifications for... [19:00:44] 10Release-Engineering-Team, 10Electron-PDFs, 10Proton, 10Readers-Web-Backlog, and 2 others: [subtask] How should we get Chromium for use in puppeteer? - https://phabricator.wikimedia.org/T178570#3697663 (10bmansurov) [19:00:52] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: [subtask] Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3697664 (10bmansurov) [19:01:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:06:30] no_justification i've installed the lfs plugin now :) [19:06:40] * paladox is figuring out how to configure it for the project now [19:07:34] (03PS3) 10Hashar: Build docker images on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) [19:07:36] (03PS1) 10Hashar: docker: build.py always exited 0 [integration/config] - 10https://gerrit.wikimedia.org/r/385225 [19:09:00] (03PS2) 10Hashar: docker: build.py always exited 1 [integration/config] - 10https://gerrit.wikimedia.org/r/385225 [19:09:03] (03CR) 10Hashar: [C: 032] docker: build.py always exited 1 [integration/config] - 10https://gerrit.wikimedia.org/r/385225 (owner: 10Hashar) [19:12:30] (03Merged) 10jenkins-bot: docker: build.py always exited 1 [integration/config] - 10https://gerrit.wikimedia.org/r/385225 (owner: 10Hashar) [19:16:51] (03CR) 10Hashar: [C: 04-1] "Given how long it takes, most probably that should just poll the SCM and report somewhere instead of being triggered on each merge." [integration/config] - 10https://gerrit.wikimedia.org/r/385213 (https://phabricator.wikimedia.org/T178594) (owner: 10Hashar) [19:19:11] Yippee, build fixed! [19:19:11] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #168: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/168/ [19:23:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10netops, 10Jenkins: Disable Jenkins autodiscovery system - https://phabricator.wikimedia.org/T178608#3697773 (10hashar) a:03hashar [19:36:01] hey yall, trying to set up gerrit + jenkins for a python project, not finding many docs [19:36:04] tips? [19:39:26] ottomata hi, see https://gerrit-review.googlesource.com/Documentation/linux-quickstart.html [19:39:35] it's new doc's for new users :) [19:39:48] hm? sorry, i just want to enable CI on my python repo :) [19:40:02] oh, on gerrit.wikimedia.org? [19:40:13] integration/config is the repo you want :) [19:40:36] k ya looking there now, not totally clear but i think i get it [19:43:05] woo hoo [19:43:09] i think i got lfs working [19:43:15] no_justification :) [19:43:24] at least it's showing on https://gerrit2.git.wmflabs.org/r/#/admin/projects/test/test [19:43:29] (03PS1) 10Ottomata: Enable tox for python cergen [integration/config] - 10https://gerrit.wikimedia.org/r/385237 (https://phabricator.wikimedia.org/T166167) [19:43:38] paladox: ^ [19:43:39] ? [19:43:49] yep [19:43:52] that's correct [19:43:52] :) [19:44:08] (03CR) 10Paladox: [C: 031] Enable tox for python cergen [integration/config] - 10https://gerrit.wikimedia.org/r/385237 (https://phabricator.wikimedia.org/T166167) (owner: 10Ottomata) [19:44:17] if I merge that everything will be ok, right? :) [19:44:46] yes [19:44:50] i see no syntax error [19:45:10] ok found https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Deploy_configuration [19:45:11] :) [19:45:20] :) [19:46:59] (03CR) 10Ottomata: [C: 031] Enable tox for python cergen [integration/config] - 10https://gerrit.wikimedia.org/r/385237 (https://phabricator.wikimedia.org/T166167) (owner: 10Ottomata) [19:48:05] hmm,m i guess i can't merge! [19:48:22] oh well, i can wait :) [19:48:58] Locking support detected on remote "origin". Consider enabling it with: [19:48:58] $ git config lfs.https://Paladox@gerrit2.git.wmflabs.org/r/a/test/info/lfs.locksverify true [19:48:58] Git LFS: (1 of 1 files) 46 B / 46 B [19:48:59] :) [19:50:18] note locks are only supported from 2.14 [19:50:32] in the lfs plugin [19:53:35] https://gerrit.wikimedia.org/r/#/c/385239/ [19:59:39] no_justification im not sure weather we should enable lfs for all mediawiki/* [19:59:41] ? [20:19:11] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 3 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3697944 (10Paladox) I have tested the plugin. The configuration above should work ^^ :). then we have to initialise the re... [20:41:51] Yippee, build fixed! [20:41:52] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #552: 09FIXED in 50 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/552/ [20:41:55] 10Continuous-Integration-Infrastructure (shipyard): operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3697991 (10hashar) [20:42:49] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #552: 04FAILURE in 1 min 48 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/552/ [20:43:40] 10Continuous-Integration-Infrastructure (shipyard): operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3698003 (10hashar) [20:55:44] 10Continuous-Integration-Infrastructure (shipyard): operations-puppet Docker container takes a while to build - https://phabricator.wikimedia.org/T178620#3698035 (10hashar) [21:10:49] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:25:40] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:28:17] 10Continuous-Integration-Config, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017), 10Patch-For-Review: Run MediaWiki tests on PHP 7 - https://phabricator.wikimedia.org/T144962#3698110 (10CCicalese_WMF) [21:29:01] paladox: No. I'd rather do it on a repo-by-repo basis [21:29:10] ok [21:29:19] I'd rather have a good audit trail of /what/ is checking in large files, ya know? [21:29:25] yeh [21:29:31] the patch i uploaded did it for the first repo [21:29:43] we have to do something like ^ [21:31:20] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3698128 (10CCicalese_WMF) [21:35:49] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:55:39] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures