[00:06:46] 10Deployment-Systems, 3Scap3: Scap3 targets should use a config file rather than `key:value` arguments - https://phabricator.wikimedia.org/T116432#1755962 (10dduvall) [00:09:37] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1755973 (10Catrope) >>! In T116447#1750221, @jcrespo wrote: >> Good, that this happens only at beta. Imagine that would happend at prod... [00:10:46] bd808, can you give an example of using -i argument to sync-common please? [00:14:35] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: MediaWiki CI tests should run with allow_url_fopen set to false - https://phabricator.wikimedia.org/T116704#1755977 (10saper) [00:50:19] thcipriani: Did we ever port our "autosign puppet and salt" stuff to beta so instances at least start sanely? [01:05:49] i don't suppose there is an already defined way to run `hhvm -m debug --debug-host ` in beta to step through a web request? [01:06:00] i can setup the hack i use in prod, but if someone already set up something legit would be nice :) [01:06:55] (fwiw, that hack is to boot an hhvm instance on alternate ports and then use a fastcgi client library i wrote to send hand crafted requests at it) [01:10:03] Project beta-scap-eqiad build #76038: 04FAILURE in 0.65 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76038/ [01:10:09] That's me ^ [01:16:16] Yippee, build fixed! [01:16:16] Project beta-scap-eqiad build #76039: 09FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76039/ [01:21:03] Weird. [01:21:35] ostriches: I want to say that we did do the autosign puppet and salt thing on beta for new instances, but I'm not 100% [01:21:46] I think so, it's working at least. [01:21:55] Tho, I'm curious where my local scap hack went. [01:22:01] Build should not have fixed itself :p [01:22:51] huh, that is weird. nothing I know of that auto-anythings scap on beta. [01:23:45] probably user error [01:23:47] i should eat. [01:24:52] that also happened earlier today (11:08 sf time) [01:26:45] scap being broken or scap fixing itself? Or both of those things? [01:26:50] fixing itself [01:27:12] (so i guess both) [01:27:27] actually maybe just fixed, i dont see the break message [01:27:42] yeah, I think that was the same problem with the localization cache that I saw a few weeks back. [01:28:50] I need to investigate that further. I thought it was just a one-time localization cache being zonked for some unknown reason thing, but I guess it's more of an actual thing than I thought initially. [01:29:59] 10Deployment-Systems, 3Scap3, 10Analytics, 6Services, 6operations: Use Scap3 for deploying AQS - https://phabricator.wikimedia.org/T114999#1756099 (10Dzahn) [01:32:11] It broke when I ran it by hand from deployment-bastion. [01:32:20] But worked when jenkins did its shiz. [01:32:24] And my commit doesn't seem to be there. [01:34:51] Ah, wrong branch [01:35:20] Project beta-scap-eqiad build #76041: 04FAILURE in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76041/ [01:35:31] Ok, good [01:35:34] That's what I wanted. [01:37:01] (03PS5) 10Chad: [WIP] Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [01:40:42] Krenair: it's used internally by the sync-* commands. One example is https://github.com/wikimedia/mediawiki-tools-scap/blob/master/scap/main.py#L400-L403 [01:45:55] Yippee, build fixed! [01:45:56] Project beta-scap-eqiad build #76042: 09FIXED in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76042/ [01:51:30] bd808, thanks [01:52:08] bd808, I used labs vagrant for the first time earlier, and noticed it tries to use mediawiki-vagrant.wmflabs.org [01:52:26] it does? [01:52:44] did for me [01:52:56] it should use the hostname that is deployed on +.wmflabs.org [01:53:18] think it was ::mediawiki::server_url or one of the restbase config settings that did it [01:53:30] I changed a few things in puppet and got it working well enough to do what I wanted [01:53:36] still seemed strange [01:54:15] sounds like something that hardcoded the wrong thing in hiera maybe [01:54:31] Meh, making a new deployment::server in beta is proving annoying :( [01:54:33] I haven't tried using restbase or ve on a labs hos [01:54:44] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter validate_cmd at /etc/puppet/modules/nutcracker/manifests/init.pp:58 on node deployment-mira.deployment-prep.eqiad.wmflabs [01:54:44] ostriches: :(( it is a bit of a pain [01:55:05] If I could figure out why that's not working, I'd probably be set. [01:55:07] there is already a mira in prep [01:55:30] That's what I say to myself every time I deal with puppet, ostriches. [01:55:46] Ah, there be! [01:55:55] Although it's named wrongly :p [01:55:59] I built it to test that patch you've been rebasing [01:56:24] Krenair: :P I didn't know that the project subdomain was just a fake thing [01:56:45] it was the first host I built after that was made the default in the fqdns [01:56:50] You've been stuck in too many relatively-tidy-ish labs projects. [01:58:33] huh, that is weird. nothing I know of that auto-anythings scap on beta. [01:58:41] jenkins [01:58:51] That's funny, because it keeps checking out bd808's non-branch :p [01:59:32] ostriches: the trebuchet package acting up? [01:59:44] puppet makes that happen [02:00:15] puppet has trebuchet fetch the latest tag from the deploy server [02:01:05] a git deploy from bastion should fix things [02:01:49] Krenair: if you have the puppet hacks you needed for mw-vagrant on labs to work right I'd be glad to review and try to get it fixed for everyone [02:02:29] even just a paste of the git diff might tell me what to look for [02:06:14] Project beta-scap-eqiad build #76044: 04FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76044/ [02:38:44] bd808: It's mostly working, minus our permission cruft we've accumulated over the years. [02:39:12] Which I'm taking the approach of "everything in -staging should be g+w :wikidev" [02:39:26] I think that's right too [02:59:57] 02:56:06 Started sync-masters [02:59:57] sync-masters: 100% (ok: 1; fail: 0; left: 0) [02:59:57] 02:57:42 Finished sync-masters (duration: 01m 36s) [03:00:01] bd808: ^^ :D [03:00:28] excellent [03:00:40] just need some chmod love? [03:00:53] s/chmod/chown/ probably [03:01:31] We need to chgrp everything to wikidev, then chmod g+w [03:03:40] cache/l10n might be an issue... [03:04:19] It might need to be excluded from the master->master sync [03:06:20] Yippee, build fixed! [03:06:21] Project beta-scap-eqiad build #76050: 09FIXED in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76050/ [03:09:27] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #761: 04FAILURE in 19 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/761/ [03:14:34] bd808: Now we just need CR on puppet :p [03:16:30] Project beta-scap-eqiad build #76051: 04FAILURE in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76051/ [03:18:14] rsync: failed to set times on "/srv/mediawiki-staging/.": Operation not permitted (1) [03:18:18] silly rsync. [03:18:39] why not? it's group writable. [03:18:48] and owned by root:wikidev which is odd but w/e. [03:18:52] Maybe puppet's fault. [03:19:38] what's the ownership on tin? [03:20:09] Same [03:20:21] I suppose the other thing we could do is run the rsync as root instead... [03:20:32] that would fix all of the permissions problems at once [03:20:57] If we did everything as root.... ;-P [03:21:19] we are using the module definition so we wouldn't be able to get random files from the other deploy server [03:22:53] we could craft the sudoers entry to only allow using the exact command that scap would generate [03:23:08] but yeah it would be nicest not to resort to that [03:25:41] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #861: 04FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/861/ [03:48:44] bd808: undeployed you for the evening. it "works" but until we sort out the permissions, it'll keep jenkins yelling. [03:48:49] I shall revisit tomorrow. [03:48:50] :) [03:49:13] progress! [03:49:32] I pretty much gave up on that patch out of ... boredom I guess [03:56:02] Yippee, build fixed! [03:56:03] Project beta-scap-eqiad build #76055: 09FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76055/ [04:24:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #604: 04FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/604/ [05:19:00] 10Beta-Cluster-Infrastructure: 404 handler for bits.beta.wmflabs.org serves unparsed php - https://phabricator.wikimedia.org/T76810#1756271 (10bd808) 5Open>3Resolved a:3bd808 >>! In T76810#1253896, @Krinkle wrote: > I can still reproduce this. Seems to be fixed now. Not sure what changed since May. [05:36:25] 10Deployment-Systems, 3Scap3: Scap3 targets should use a config file rather than `key:value` arguments - https://phabricator.wikimedia.org/T116432#1756303 (10mmodell) I was originally concerned about hitting the limit for maximum command line length, however, it appears that on Linux that limit is 2090327 byte... [07:15:13] 10MediaWiki-Codesniffer: Position of boolean operators inside an if condition - https://phabricator.wikimedia.org/T116561#1756399 (10TasneemLo) [09:08:56] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1756500 (10Joe) @bd808 I will work on the wikitech settings right away, and I'll take a look at the other files as well. [09:17:24] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1756503 (10Joe) And btw yes - I think we could anyways move to use mira for the time being instead of tin. [09:32:37] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 7HHVM, 5Patch-For-Review: Reimage mw1152 as a terbium replacement - https://phabricator.wikimedia.org/T116728#1756538 (10Joe) 3NEW a:3Joe [09:38:06] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1756557 (10hashar) You might want to follow up on query optimization and code audit in a different task? [09:47:35] 10Continuous-Integration-Config, 10Dumps-Generation, 6operations, 5Patch-For-Review, 7WorkType-Maintenance: operations/dumps repo should pass flake8 - https://phabricator.wikimedia.org/T114249#1756583 (10ArielGlenn) [09:48:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: MediaWiki CI tests should run with allow_url_fopen set to false - https://phabricator.wikimedia.org/T116704#1756585 (10hashar) You probably want to disable it explicitly in `tests/phpunit/MediaWikiTestCase.php` setUp() method. Haven't tested the... [10:00:00] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 2 others: ContentTranslation zend tests fail because of a missing Wikidata/Scribunto dependency - https://phabricator.wikimedia.org/T116732#1756611 (10Amire80) 3NEW [10:00:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of a missing Wikidata/Scribunto dependency - https://phabricator.wikimedia.org/T116732#1756611 (10Amire80) [10:01:20] (03PS6) 10Paladox: Add new extension-unittests-composer template [integration/config] - 10https://gerrit.wikimedia.org/r/247920 (https://phabricator.wikimedia.org/T90303) [10:08:34] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 6Discovery: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#1756644 (10Deskana) Is this still an issue? [10:17:59] hashar: could you review https://gerrit.wikimedia.org/r/#/c/247920/ please. [10:21:56] (03PS7) 10Paladox: Update extensions using the deprecated qunit test [integration/config] - 10https://gerrit.wikimedia.org/r/245495 [10:22:25] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 6Discovery: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#1756695 (10hashar) 5Open>3Resolved a:3hashar Using http://graphite.wmflabs.org/ we have a few metrics under `BetaMediaWiki.CirrusSearch.requestTime.` For t... [10:33:26] (03PS6) 10Hashar: Convert 'operations-puppet-doc' job to run on a labs slave [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [10:34:32] (03CR) 10Hashar: Convert 'operations-puppet-doc' job to run on a labs slave (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [10:58:44] (03PS7) 10Hashar: Convert 'operations-puppet-doc' job to run on a labs slave [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [11:06:05] (03PS8) 10Hashar: Convert 'operations-puppet-doc' job to run on a labs slave [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [11:07:11] (03CR) 10Hashar: [C: 032] "Job updated and tested." [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [11:21:36] (03Merged) 10jenkins-bot: Convert 'operations-puppet-doc' job to run on a labs slave [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [11:23:18] 10Continuous-Integration-Config, 5Patch-For-Review: https://doc.wikimedia.org/puppet/ does not purge obsolete class documentation - https://phabricator.wikimedia.org/T96576#1756833 (10hashar) Might be fixed by https://gerrit.wikimedia.org/r/204982 which migrated the operations-puppet-doc job and thus use rsync... [11:23:27] 10Continuous-Integration-Config: https://doc.wikimedia.org/puppet/ does not purge obsolete class documentation - https://phabricator.wikimedia.org/T96576#1756835 (10hashar) [11:24:09] 10Continuous-Integration-Infrastructure: Migrate all jobs to labs slaves - https://phabricator.wikimedia.org/T86659#1756838 (10hashar) [11:24:49] 10MediaWiki-Codesniffer, 3Outreachy-Round-11: Outreachy proposal for : Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T115585#1756839 (10TasneemLo) [11:25:18] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1756840 (10Tobi_WMDE_SW) @JanZerebecki AFAIK it uses Saucelabs whenever SAUCE_ONDEMAND_USERNAME and SAUCE_ONDEMAND_ACCESS_KEY are set. Unsetting these should ma... [11:49:12] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of a missing Wikidata/Scribunto dependency - https://phabricator.wikimedia.org/T116732#1756884 (10hashar) Wikibase tests that depends on... [11:49:41] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of a missing Wikidata/Scribunto dependency - https://phabricator.wikimedia.org/T116732#1756885 (10hashar) [11:49:56] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of Wikibase\Test\EditEntityTest::testAttemptSaveRateLimit - https://phabricator.wikimedia.org/T116732#1756886 (10hashar) [11:52:11] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of Wikibase\Test\EditEntityTest::testAttemptSaveRateLimit - https://phabricator.wikimedia.org/T116732#1756888 (10hashar) Per IRC might fi... [11:52:55] 10Deployment-Systems, 3Scap3: Scap3 targets should use a config file rather than `key:value` arguments - https://phabricator.wikimedia.org/T116432#1756895 (10mobrovac) I'd be in favour of using a config file. In order not to use temp files, we could leverage the built-in HTTP server that is planned and serve t... [11:56:44] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of Wikibase\Test\EditEntityTest::testAttemptSaveRateLimit - https://phabricator.wikimedia.org/T116732#1756904 (10hashar) Did not fix it,... [12:58:13] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1757055 (10hashar) Don't we want jobs for all the limn data repositories? Currently we have the following Gerrit proj... [13:00:50] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #829: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/829/ [13:08:59] 10Browser-Tests, 10Continuous-Integration-Config: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1757074 (10hashar) [13:09:01] 10Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review: Define a CI entry point for Ruby code - https://phabricator.wikimedia.org/T104024#1757071 (10hashar) 5Open>3Resolved {done} see above for reference implementation that has been applied to `mediawiki/vagrant` with https://gerrit.wikimed... [13:17:13] (03PS1) 10Phedenskog: Correct path to the runner [integration/config] - 10https://gerrit.wikimedia.org/r/249112 [13:18:51] (03PS1) 10Hashar: Switch mediawiki/vagrant to use rake/Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/249113 (https://phabricator.wikimedia.org/T114860) [13:20:28] (03CR) 10Hashar: [C: 032] Switch mediawiki/vagrant to use rake/Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/249113 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [13:22:24] (03Merged) 10jenkins-bot: Switch mediawiki/vagrant to use rake/Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/249113 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [13:55:38] 10Continuous-Integration-Config, 5Patch-For-Review: Move Bundler Jenkins jobs to Nodepool instances - https://phabricator.wikimedia.org/T114860#1757200 (10hashar) From T104024 the entry point is described at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby mediawiki/vagrant is migrated [14:28:33] (03PS1) 10Hashar: mwext-Nonlinear-jslint is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/249131 (https://phabricator.wikimedia.org/T63614) [14:28:39] 10Continuous-Integration-Infrastructure, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1757296 (10hashar) [14:28:44] (03CR) 10Hashar: [C: 032] mwext-Nonlinear-jslint is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/249131 (https://phabricator.wikimedia.org/T63614) (owner: 10Hashar) [14:29:46] (03Merged) 10jenkins-bot: mwext-Nonlinear-jslint is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/249131 (https://phabricator.wikimedia.org/T63614) (owner: 10Hashar) [14:30:25] 10Deployment-Systems: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#1757308 (10hashar) [14:30:34] 10Deployment-Systems: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#1636723 (10hashar) s/Trebuchet/scap3/ [14:31:23] 5Continuous-Integration-Scaling, 10Deployment-Systems, 6Labs, 10Labs-Infrastructure, and 2 others: Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T111925#1757315 (10hashar) 5Open>3declined a:3hashar Solved by using a git clone directly w... [14:33:44] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 5Patch-For-Review: Fix or delete failing Flow browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94153#1757325 (10hashar) [14:33:46] 10Beta-Cluster-Infrastructure, 7Blocked-on-RelEng, 10Continuous-Integration-Config, 10Parsoid, and 2 others: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1757323 (10hashar) 5Open>3Resolved After a... [14:38:39] 10Deployment-Systems: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#1757354 (10demon) The git repositories on tin.eqiad.wmnet under /srv/deployment are never repacked or garbage collected. Yes they are, git does gc on repositories from time to time... [14:40:21] hashar: We could possibly just tweak the gc config so it happens more often :) [14:40:38] ostriches: I am not sure it actually happens :D [14:40:46] Of course it does [14:41:13] I should have taken traces, but I am pretty sure I filled that tasks because we had year olds refs and objects [14:41:31] git gc only runs on write operations doesn't it ? [14:41:49] `git gc` isn't going to prune refs and old objects. [14:41:53] I mean if you only git pull in the working directory, git gc is notgoing to run [14:42:56] and yeah the idea was from time to time to (a) get rid of loose objects, (b) repack objects in a single pack file (c) pack references in a single file :D [14:43:02] maybe I should rephrase the task? [14:44:58] git gc should handle pruning based on whatever pruneExpire is set to (default 60 days iirc) [14:45:09] 10Continuous-Integration-Infrastructure: Add pre-commit hook to check executable rights - https://phabricator.wikimedia.org/T70902#1757378 (10hashar) 5Open>3declined a:3hashar [14:45:28] And packRefs is true by default [14:46:03] We could get away with something like `git repack -adf --depth=250 --window=250` [14:47:07] Play with --depth and --window a tad. [14:48:29] "Note that users fetching over dumb protocols will have to fetch the whole new pack in order to get any contained object, no matter how many other objects in that pack they already have locally." [14:48:51] Not a huge deal, since we rsync for transport and don't plan to git for transport [14:48:54] But still, ouch. [14:49:29] ohh [14:49:29] hashar: This is a old but good read on the topic, btw. https://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/ [14:49:42] https being a dumb protocol? [14:50:44] Not necessarily. [14:50:50] something that is annoying for CI is that cloning mediawiki/core from Gerrit using https takes several minutes [14:51:00] https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols describes it. [14:51:00] and apparently a good chunk of it is rebuilding the deltas [14:51:19] i thought it could just grab the whole pack file from Gerrit instead of rebuilding it [14:52:10] * hashar reads [14:56:06] hashar: Honestly we should probably keep a copy on the CI nodes, and do `git clone --reference=/path/to/local/copy --dissociate` [14:56:31] Then it'll just do disk copies of most of the data and only fetch over the wire what it needs, then forget the reference exists at all after it's done building a clone. [14:56:49] yeah [14:56:52] gave it a try a while back [14:57:03] --dissociate is kinda new, iirc. [14:57:04] the Jenkins git plugin horribly choked when using reference :/ [14:57:57] (also, I should tweak checkoutMediaWiki to do the same in prod, it'd save a fair bit of time on the weekly train deploys) [15:00:39] hashar: What would save /even more/ space would be if we just had a single bare git repo for MW deployments and all our working copies --reference'd that :p [15:00:56] yup :-} [15:01:32] Which I kinda poked at, we could probably do, but I'm not sure how it'll fit vis à vis scap3 and MW train deploys yet. [15:02:23] hashar: You caught me in a trap before I had my coffee even! [15:02:52] I can talk about git internals too easily :p [15:04:36] (03PS6) 10Chad: Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [15:04:41] ostriches: for Nodepool I end up keeping some mirror of some repositories [15:04:49] (03CR) 10Chad: [C: 032] Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [15:04:57] so we can git clone from them and since they are on the same disk that should produces hardlinks [15:05:13] also Erik asked to get core fetched when testing operations/mediawiki-config [15:05:35] Ugh, I saw that. [15:06:06] bd808: landed co-master work. Disabled on beta *atm* until we finish sorting out permissions after my coffee. [15:06:22] (landed scap side at least, I need to find an opsen to bribe for the puppet bit) [15:06:37] thcipriani, twentyafterfour: Also cc ^ [15:06:50] (03Merged) 10jenkins-bot: Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [15:08:35] !log deploying master of scap to beta [15:08:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:08:39] ostriches: awesome! I know that's been sitting a while. [15:09:14] ostriches: Yuvi and Dzahn are usually helpful for scap puppet patches [15:09:27] Yeah, I was gonna poke dzahn [15:12:13] 5Continuous-Integration-Scaling, 7Tracking: Investigate using a cache store/restore system for package managers - https://phabricator.wikimedia.org/T116017#1757541 (10hashar) Namespacing the caches per repo/branch would causes us to have a lot of different caches which would consume a good chunk of disk space... [15:21:38] 6Release-Engineering-Team: Add all of the KPIs to a single dashboard - https://phabricator.wikimedia.org/T108744#1529356 (10hashar) [15:21:40] 6Release-Engineering-Team: Create a Releng dashboard on Grafana - https://phabricator.wikimedia.org/T110313#1757580 (10hashar) 5Open>3Resolved We had Grafana upgraded to 2.x . The dashboard https://grafana.wikimedia.org/#/dashboard/db/releng-main-page now dynamically list any dashboard tagged with `releng`... [15:22:32] 6Release-Engineering-Team: Add all of the KPIs to a single dashboard - https://phabricator.wikimedia.org/T108744#1529356 (10hashar) You can get them added to https://grafana.wikimedia.org/#/dashboard/db/releng-main-page It currently dynamically list all dashboards tagged `releng`. We can add a row that will co... [15:23:02] 6Release-Engineering-Team: Create RelEng KPI Dashboard - https://phabricator.wikimedia.org/T108745#1757589 (10hashar) [15:23:03] 6Release-Engineering-Team: Add all of the KPIs to a single dashboard - https://phabricator.wikimedia.org/T108744#1757590 (10hashar) [15:36:45] (03PS2) 10BryanDavis: Provide scap control server FQDN to proxy sync commands [tools/scap] - 10https://gerrit.wikimedia.org/r/247965 [15:37:39] bd808, so we need to get that in, run the sync to mira again and then we can deploy from mira? [15:37:50] (03PS3) 10BryanDavis: Provide scap control server FQDN to proxy sync commands [tools/scap] - 10https://gerrit.wikimedia.org/r/247965 [15:38:29] Krenair: yeah. ostriches is working on the details I think. [15:39:23] greg-g, "The SWAT deployer does their own review of the patch before committing and deploying."? [15:41:37] Krenair: you do don't you? :) [15:41:48] you review it before you merge and deploy, right? [15:42:21] to varying degrees of detail [15:42:26] that is your job [15:42:38] However, the SWAT team are not responsible for code review. [15:42:57] no, that's not what I said [15:43:04] see the previous 3 steps [15:43:05] :) [15:43:11] (or 2, I forget myemail now) [15:43:58] Krenair, bd808: Yeah, it should work now. We just need configuration for it and to sort out any permission problems along the way. [15:44:24] I disabled it in beta via hiera until I finish my coffee so I can work on permission bits. Also need to poke ops about prod merges for puppetz. [15:44:39] what needs to be done on the puppet side? [15:44:56] ostriches: we will need https://gerrit.wikimedia.org/r/#/c/247965/ as well to start a deploy from mira [15:44:57] and I guess we need to get the new version of scap itself deployed somehow [15:45:06] did that get cleaned up? [15:45:15] scap is deployed with trebuchet :) [15:45:22] bd808: shush [15:45:29] but it's true! [15:45:37] and it works [15:45:39] this doesn't answer my question.. [15:45:41] I know, it's just... it makes me smile and frown simultaneously [15:45:44] 'works' [15:45:50] "works" [15:45:55] bd808: Yeah I saw 247965, I just hadn't tested it yet. [15:46:18] * greg-g reads more backscroll before when Krenair pinged me [15:46:26] Krenair: For puppet, https://gerrit.wikimedia.org/r/#/c/224829/ [15:46:28] Just that. [15:46:39] trebuchet's problems are related to long running processes that salt loses track of (eg service restarts). It works like a charm for a simple git fetch like the scap deploys [15:46:57] Getting ops to merge that is going to be... [15:46:59] fun [15:47:18] Krenair: Filippo already mostly approved it. [15:47:33] I'm going to guess the part he didn't approve was the sudo change? [15:47:50] Just disagreement over whether we do it like that or we wrap it in a simple bash script [15:47:53] which is the most necessary part [15:48:16] The actual cmd I don't think he has a problem with [15:48:33] wrapping the rsync in a bash script would make no sense at all [15:48:54] I think his thought would be it's easier to read/audit? I dunno [15:48:55] but I think there may be a lack of general understanding of scap in the ops team [15:49:02] (and all other deploy tools) [15:49:08] and all other.... yeah... [15:53:41] 6Release-Engineering-Team: Add all of the KPIs to a single dashboard - https://phabricator.wikimedia.org/T108744#1757746 (10greg) The KPI one is at: https://grafana.wikimedia.org/dashboard/db/releng-kpis It only has the log errors one for now until we finalize the CI wait time one. [15:54:40] bd808: also just fyi, if you have any future patches for scap, we're using differential now :) [15:54:54] We can land the ones that exist already via gerrit, but in the future, new hotness! [15:55:23] which has totally been well announced and has a nice tutorial at ...??? ;) [15:55:33] bd808: dogfooding [15:55:39] ie: pre-all the niceness :) [15:55:57] we're finding the "oh shit, no one is going to like this part, we better fix it" things [15:56:09] *nod* [15:56:27] we, unfortunately, have things to fix :) [15:56:45] wouldn't it make you more nervous if we didn't? [15:56:52] I think the consensus so far is "Differential is very nice for CR, but `arc land` does some funky things" [15:57:00] thcipriani: true [15:57:13] yup, what the bird said [15:57:31] how do I subscribe to all proposed patches in differential against a repository? [15:57:44] Krenair: I created a herald rule [15:57:56] Would be my guess ^ [15:57:57] herald rules are kinda like that wiki page for gerrit reviews [15:58:12] I can't tell whether that's terrible or good. [15:58:17] good [15:58:22] herald rules are pretty powerful [15:58:37] can normal people make herald rules now? [15:58:39] Actually, there's a feature in gerrit that does the exact same thing as the wiki page + bot :p [15:58:40] and they're built in, not a third-party bot or anything [15:58:56] bd808: yeah, "personal" ones, not "global" ones, but they do what you need them to do [15:59:10] ostriches: fine fine, then it's better than the cludgey system we set up :) [15:59:14] oh cool. I must have missed that announcement [15:59:34] team weekly time [16:00:13] Oh man it is that time [16:03:14] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:14] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:13:16] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1757852 (10hashar) Should use the Jessie operating system. [16:13:30] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1757855 (10RobH) [16:24:44] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 7HHVM, 5Patch-For-Review: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1757899 (10Joe) The ldap settings file has now permissions that should allow scap not to choke on it. How... [16:25:47] ostriches: ^ _joe_ looks like your man for the puppet merges [16:29:02] 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 7HHVM, 5Patch-For-Review: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1757919 (10bd808) >>! In T87036#1757899, @Joe wrote: > The ldap settings file has now permissions that sho... [16:29:27] (03PS4) 10BryanDavis: Provide scap control server FQDN to proxy sync commands [tools/scap] - 10https://gerrit.wikimedia.org/r/247965 (https://phabricator.wikimedia.org/T104826) [16:36:33] 5Continuous-Integration-Scaling, 6operations, 7Blocked-on-Operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1757948 (10fgiunchedi) I believe package_builder will try importing from jessie-wikimedia if WIKIMEDIA=yes is passed o... [16:59:36] (03PS1) 10Arlolra: Point parsoid jobs to bin/ [integration/config] - 10https://gerrit.wikimedia.org/r/249163 [17:01:03] 10MediaWiki-Codesniffer: phpcs: Enforce clone is not a function - https://phabricator.wikimedia.org/T116779#1758147 (10Umherirrender) 3NEW [17:10:32] 5Continuous-Integration-Scaling, 6operations, 7Blocked-on-Operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1758179 (10hashar) `WIKIMEDIA=yes` does not include either `jessie-backports` or `jessie-wikimedia/thirdparty`. We wo... [17:13:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of Wikibase\Test\EditEntityTest::testAttemptSaveRateLimit - https://phabricator.wikimedia.org/T116732#1758194 (10Amire80) Appears to be f... [17:15:02] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1758218 (10dduvall) >>! In T116447#1755973, @Catrope wrote: > Do we have a way to find out what the slow query was that caused this out... [17:16:43] (03PS2) 10Krinkle: Correct path to the runner [integration/config] - 10https://gerrit.wikimedia.org/r/249112 (owner: 10Phedenskog) [17:17:34] (03PS3) 10Krinkle: Correct path to the runner [integration/config] - 10https://gerrit.wikimedia.org/r/249112 (owner: 10Phedenskog) [17:19:06] (03CR) 10Krinkle: [C: 032] "Compiling and deploying now.." [integration/config] - 10https://gerrit.wikimedia.org/r/249112 (owner: 10Phedenskog) [17:23:58] 10MediaWiki-Codesniffer: SpaceyParenthesis bug - https://phabricator.wikimedia.org/T116519#1758261 (10Krinkle) [17:24:26] (03Merged) 10jenkins-bot: Correct path to the runner [integration/config] - 10https://gerrit.wikimedia.org/r/249112 (owner: 10Phedenskog) [17:43:49] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1758398 (10dduvall) It appears there was a query logged by HHVM's SlowTimer 5 minutes prior to moment I killed the other one, in case t... [18:01:24] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1179083 (10RobH) [18:01:26] 5Continuous-Integration-Scaling, 10hardware-requests, 6operations: eqiad: 2 hardware access request for CI isolation on labsnet - https://phabricator.wikimedia.org/T93076#1758535 (10RobH) 5Open>3Resolved I'm resolving this task, as the install of scandium is tracked on T95046. [18:03:40] thcipriani: Are you running the train today? Do you know when wmf.4 will be cut? I want something to definitely not get into it. :-) [18:04:01] James_F: twentyafterfour is typically the train-runner. [18:04:49] Aha, OK. [18:04:58] (And yes, you're right, I'd forgotten. :-)) [18:05:03] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1758560 (10RobH) [18:05:10] he usually starts around now-ish fwiw (but that may or may not be the case) [18:05:22] * James_F blames DST. :-) [18:05:48] 10Beta-Cluster-Infrastructure: Investigate slow query logging/digest for Beta Cluster - https://phabricator.wikimedia.org/T116793#1758568 (10dduvall) 3NEW [18:05:52] 10Beta-Cluster-Infrastructure, 6Labs, 10Salt: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#1758577 (10ArielGlenn) [18:06:40] (03CR) 10Subramanya Sastry: [C: 031] Point parsoid jobs to bin/ [integration/config] - 10https://gerrit.wikimedia.org/r/249163 (owner: 10Arlolra) [18:17:57] James_F: I'm cutting the branch right now [18:18:22] twentyafterfour: Does that mean we can merge to MW-core/MW-vendor master now without it getting into wmf.4? [18:18:55] James_F: wait just a bit longer, core gets branched last [18:19:00] Kk. [18:19:19] still running through all the extension branches, which takes a really long time (~30 minutes) [18:19:36] We should delete some extensions so it runs faster. [18:19:39] I'll let you know when the branch is done [18:19:51] James_F: I'd be ok with that ;) [18:19:55] * James_F grins. [18:22:28] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: Connection refused [18:28:41] James_F: core branch cut at 2cb813ff13f1b005fbb106b40fb9262430a528e1 [18:28:43] you can merge [18:31:08] actually vendor might not be cut yet.. [18:33:51] twentyafterfour: Argh. [18:34:17] vendor branched at 2df0f40378bc3e57f5bd9a567131c2dd2eb70f2d [18:34:22] Aha. [18:34:24] Thank you! [18:34:35] you're welcome [18:51:54] (03PS2) 10Legoktm: Point parsoid jobs to bin/ [integration/config] - 10https://gerrit.wikimedia.org/r/249163 (owner: 10Arlolra) [18:53:58] (03CR) 10Legoktm: [C: 032] Point parsoid jobs to bin/ [integration/config] - 10https://gerrit.wikimedia.org/r/249163 (owner: 10Arlolra) [18:55:47] (03Merged) 10jenkins-bot: Point parsoid jobs to bin/ [integration/config] - 10https://gerrit.wikimedia.org/r/249163 (owner: 10Arlolra) [19:03:15] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1758864 (10RobH) [19:07:47] 10Deployment-Systems, 3Scap3: Scap3 targets should use a config file rather than `key:value` arguments - https://phabricator.wikimedia.org/T116432#1758881 (10mmodell) @mobrovac: That was one of my main reasons for proposing the built-in HTTP server ;) I do see @dduvall's point about the arguments being in the... [19:08:28] legoktm: Did you mark https://gerrit.wikimedia.org/r/#/c/154757/ as a draft? It's linked from https://phabricator.wikimedia.org/T71685 but apparently I don't have rights to see… [19:09:14] James_F: uh, I can't see it either... [19:09:21] legoktm: That's… interesting. [19:09:51] legoktm: Do you recall off-hand whether it worked? :-) [19:10:02] wtf [19:10:05] where did it go? [19:10:16] Uhm. [19:10:18] Probably not [19:10:30] But I actually know what I'm doing now, so I could do it properly if people are interested [19:11:10] I am for VE, if nothing else. [19:11:14] VE-MW, that is. [19:11:28] And Daniel Kinzler and I were just talking about how he wanted it for Wikibase. [19:11:57] Does VE have phpunit tests? [19:12:30] Hmmm, Wikibase would be fun. [19:12:36] mwext-VisualEditor-qunit yes. [19:12:40] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1758928 (10RobH) [19:12:44] Though mostly we care about the JS coverage. [19:13:01] Oh that patch was for phpunit coverage [19:13:06] I haven't looked how to do qunit coverage [19:13:15] mwext-VisualEditor-qunit I guess could be switched over to use karma-coverage instead of just karma, right? [19:13:19] * James_F speculates wildly. [19:13:24] I'll create a task. [19:13:44] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1758937 (10RobH) a:5RobH>3hashar I think the service implementation of this would belong to @hashar, so I am assigning this task to him for the final steps. If this i... [19:13:56] I'm not sure, because it goes through core and stuff [19:14:00] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1758939 (10RobH) 5stalled>3Open [19:16:27] 10Continuous-Integration-Infrastructure: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma - https://phabricator.wikimedia.org/T116808#1758975 (10Jdforrester-WMF) 3NEW [19:17:26] legoktm: Ultimately the 'qunit-karma' macro just does `npm install && grunt karma:main`. [19:17:29] legoktm: So… [19:28:23] (03PS1) 10Hashar: Update rake-jessie to use 'rake' [integration/config] - 10https://gerrit.wikimedia.org/r/249219 [19:29:34] (03CR) 10Hashar: [C: 04-1] "Depends on puppet change https://gerrit.wikimedia.org/r/#/c/249213/" [integration/config] - 10https://gerrit.wikimedia.org/r/249219 (owner: 10Hashar) [19:32:23] (03PS1) 10Florianschmidtwelzow: Add GoogleAPIClient as dependency to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/249223 [19:49:31] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs - https://phabricator.wikimedia.org/T116816#1759114 (10Bawolff) 3NEW [20:11:15] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs - https://phabricator.wikimedia.org/T116816#1759203 (10Krenair) According to random production imagescaler mw1153 this is supposed to come from package `libvips-tools` [20:18:01] What host in beta would be responsible for image scaling? [20:19:06] Krenair: that is routed via deployment-upload which is outdated / unpuppetized [20:19:16] Krenair: https://phabricator.wikimedia.org/T84950 [20:19:39] not Deployment-tmh01.deployment-prep.eqiad.wmflabs ? [20:19:53] so your vips task T116816 should probably be blocked by https://phabricator.wikimedia.org/T84950 [20:19:56] I know brion was doing some fun stuff there for a while [20:20:06] greg-g, that'd be video scaling [20:20:19] wasn't sure if they co-located or nott [20:20:20] -t [20:20:34] I think the idea is more or less something like: [20:20:38] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs - https://phabricator.wikimedia.org/T116816#1759263 (10Krenair) -> {T84950} [20:20:49] upload -> upload cache -> deployment-upload -> text cache -> app server [20:21:07] and I guess the app server do not have the libvips-tools [20:21:29] so yeah there is a bunch of refactoring to happen in the way we deal with thumbnails on beta [20:21:40] the arch is so scary I don't want to put my fingers into it [20:22:45] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [20:28:29] :/ [20:29:44] 10Deployment-Systems, 6operations: Investigate whether mod_dav needs to stay enable on tin/terbium - https://phabricator.wikimedia.org/T116823#1759278 (10Reedy) 3NEW [20:30:00] 6Release-Engineering-Team, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1759285 (10BBlack) >>! In T102566#1732320, @Tgr wrote: > Yes, a couple hours ago. We should write to mediawiki-... [20:30:25] 10Deployment-Systems, 6operations: Investigate whether mod_dav needs to stay enabled on tin/terbium - https://phabricator.wikimedia.org/T116823#1759287 (10Reedy) [20:31:45] !log Generating new Nodepool snapshot ( https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot ) to have 'rake' included ( puppet: https://gerrit.wikimedia.org/r/#/c/249219/ ) [20:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:40:22] !log Nodepool snapshot ci-jessie-wikimedia-1445977928 generated. Includes /usr/bin/rake ( puppet: https://gerrit.wikimedia.org/r/#/c/249219/ ) [20:40:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:42:46] !log Nuking Nodepool instances using the previous snapshot ( ci-jessie-wikimedia-1445955240 ) [20:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:48:33] 10MediaWiki-Codesniffer: Position of boolean operators inside an if condition - https://phabricator.wikimedia.org/T116561#1759337 (10Legoktm) I think you're looking for https://www.mediawiki.org/wiki/Manual:Coding_conventions#Line_continuation ? The operator should be at the beginning of the next line. [20:49:09] (03CR) 10Hashar: "Puppet change https://gerrit.wikimedia.org/r/#/c/249213/ has been merged." [integration/config] - 10https://gerrit.wikimedia.org/r/249219 (owner: 10Hashar) [20:54:50] (03CR) 10Hashar: [C: 032] "I have refreshed 'rake-jessie'. Validated it on https://gerrit.wikimedia.org/r/#/c/249145/" [integration/config] - 10https://gerrit.wikimedia.org/r/249219 (owner: 10Hashar) [20:56:48] (03Merged) 10jenkins-bot: Update rake-jessie to use 'rake' [integration/config] - 10https://gerrit.wikimedia.org/r/249219 (owner: 10Hashar) [21:20:27] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #49: 04FAILURE in 4 min 25 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/49/ [21:23:41] marxarelli: on other news the Jenkins job 'rake-jessie' now uses 'bundle exec rake' [21:23:47] it used to be 'bundle exec rake2.1' [21:23:56] so people can pin whatever rake version they want in their /Gemfile [21:24:24] i think rake2.1 is just a packing thing for systems where ruby 2 is not the default [21:24:24] I think we will look at migrating all repos having ruby to that rake-jessie job [21:24:30] not rake version 2.1 [21:24:34] yeah [21:24:44] cool! [21:24:53] on Debian the package 'ruby2.1' ships 'rake2.1' [21:25:00] and nodepool images were missing the 'rake' package [21:25:27] the ruby boilerplate is documented at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Ruby [21:25:35] maybe we will find out a way to migrate browser tests to it as well [21:26:47] hashar: i'm a little sad about the 3m builds :( [21:27:15] 3 minutes? [21:27:21] ah yeah [21:27:25] we used to have ~ 20s builds for mw-vagrant [21:27:30] on mediawiki/vagrant puppet-lint is way too slow [21:27:51] operations/puppet suffers from the same issue with puppet-lint taking up to 40 seconds iirc [21:28:06] are you sure it's not due to nodepool spin up? [21:28:11] maybe rake can run the tasks in parallel ? [21:28:31] and lack of package cache? [21:28:34] the instances are already booted so they are just consumed [21:28:41] but yeah the lack of package cache is an issue [21:28:47] as well as having to compile the native modules :-(((( [21:29:22] if you look at a build like https://integration.wikimedia.org/ci/job/rake-jessie/16/consoleFull [21:29:30] the difference between mw-vagrant unit tests with and without a warm package cache is like 1m20s vs 20s [21:29:38] on the left side bar you can change the timestamp to be relative "Elapsed Time" [21:29:39] that's backwards [21:29:57] 20s warm, 1m20s cold [21:30:15] that build is for MWV and it took 2 minutes and 20 sec to reach bundle exec rake2.1 test :( [21:30:19] the actual tests take about 1s, the rest is overhead [21:32:35] that is why I emphasis the task and QA mail about caching package managers [21:33:00] I am pretty much confident we can easily cache the tarballs. Potentially via a shared proxy [21:33:30] the native modules, I am not so sure cause the native compilation is not necessarily stored in the package manager cache :-/ [21:36:29] marxarelli: hey python world has a soft that caches binaries to S3 :-} https://pypi.python.org/pypi/pip-accel#storing-the-binary-cache-on-amazon-s3 [21:37:00] hashar: natively compiled gems should be under GEM_HOME [21:37:05] don't know about the others [21:37:22] yeah GEM_HOME [21:38:43] what I like with rubygems [21:38:57] is that you can have several versions installed in parallel [21:44:44] 6Release-Engineering-Team, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1759636 (10Tgr) [[ https://lists.wikimedia.org/pipermail/mediawiki-announce/2015-October/000183.html | Done. ]] [21:47:08] tgr: well done :-} [21:59:47] ah [22:20:41] greg-g, did you see https://gerrit.wikimedia.org/r/#/c/248639/ ? [22:24:18] Krenair: barely :) [22:45:40] 10MediaWiki-Codesniffer, 7Documentation: Document how phpcbf should be set up and used - https://phabricator.wikimedia.org/T116864#1760000 (10Legoktm) 3NEW [22:51:50] 10MediaWiki-Codesniffer, 7Easy: Update mediawiki/tools/codesniffer repository to use phpcs.xml file - https://phabricator.wikimedia.org/T116866#1760022 (10Legoktm) 3NEW [22:55:21] 10MediaWiki-Codesniffer: phpcs: Enforce clone is not a function - https://phabricator.wikimedia.org/T116779#1760038 (10Legoktm) > like there is (already active?) for require and require_once From what I can tell we don't have sniffs for this right now. I'll update this task to include that. [22:56:17] 10MediaWiki-Codesniffer: phpcs: Enforce clone, require, etc., are not functions - https://phabricator.wikimedia.org/T116779#1760052 (10Legoktm) [23:09:11] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches - https://phabricator.wikimedia.org/T90303#1760136 (10Legoktm) [23:23:46] (03PS1) 10Legoktm: build: Pass -s to phpcs for easier debugging [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/249320