[00:34:52] (03PS1) 10Awight: Experimental lint and phpunit for the SmashPig repo [integration/config] - 10https://gerrit.wikimedia.org/r/284618 [00:37:00] 10Continuous-Integration-Config, 10Fundraising-Backlog, 07FR-ActiveMQ, 07FR-Smashpig, 03Fundraising Sprint Hermit Crab Husbandry: SmashPig CI should run phpunit - https://phabricator.wikimedia.org/T133248#2226085 (10awight) [01:21:00] (03CR) 10Mattflaschen: [C: 032] "Haven't tested, since the version of mediawiki_selenium I currently have doesn't even have rake_task.rb." [selenium] - 10https://gerrit.wikimedia.org/r/282370 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [01:22:34] (03Merged) 10jenkins-bot: Skip Cucumber features or scenarios tagged with `@skip` tag [selenium] - 10https://gerrit.wikimedia.org/r/282370 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [03:13:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #949: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/949/ [03:13:48] Yippee, build fixed! [03:13:48] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1052: 09FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1052/ [04:46:47] (03PS7) 10Aashaka: Add sniff to check if boolean operator on next line inside multiline condition [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) [05:24:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #764: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/764/ [06:26:20] Yippee, build fixed! [06:26:20] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #969: 09FIXED in 7 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/969/ [08:07:07] 10Continuous-Integration-Infrastructure, 06Operations: Investigate usage of ttf-ubuntu-font-family which is not available on Jessie - https://phabricator.wikimedia.org/T103325#2226489 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [08:07:24] 10Continuous-Integration-Infrastructure, 06Operations: Investigate usage of ttf-ubuntu-font-family which is not available on Jessie - https://phabricator.wikimedia.org/T103325#1387187 (10MoritzMuehlenhoff) 05Open>03Resolved I built the trusty jessie of src:ubuntu-font-family-sources for jessie-wikimedia an... [08:20:58] Yippee, build fixed! [08:20:58] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #928: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/928/ [08:54:53] (03CR) 10Hashar: [C: 04-1] Experimental lint and phpunit for the SmashPig repo (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/284618 (https://phabricator.wikimedia.org/T133248) (owner: 10Awight) [09:09:40] (03CR) 10Hashar: [C: 04-1] "The repository does not even have a composer.json so that makes no sense to me. Additionally that extension hasn't received any useful cod" [integration/config] - 10https://gerrit.wikimedia.org/r/284492 (owner: 10Paladox) [09:12:01] (03CR) 10Hashar: [C: 04-1] "SemanticMediaWiki is primary developed on Github at https://github.com/SemanticMediaWiki/SemanticMediaWiki the Gerrit version is basicall" [integration/config] - 10https://gerrit.wikimedia.org/r/284490 (owner: 10Paladox) [09:27:36] (03PS1) 10Hashar: [JsonData] run jsonlint via npm [integration/config] - 10https://gerrit.wikimedia.org/r/284659 [09:27:53] (03CR) 10Hashar: [C: 032] [JsonData] run jsonlint via npm [integration/config] - 10https://gerrit.wikimedia.org/r/284659 (owner: 10Hashar) [09:28:59] (03Merged) 10jenkins-bot: [JsonData] run jsonlint via npm [integration/config] - 10https://gerrit.wikimedia.org/r/284659 (owner: 10Hashar) [09:30:41] (03CR) 10Hashar: "I have added npm via https://gerrit.wikimedia.org/r/#/c/284659/" [integration/config] - 10https://gerrit.wikimedia.org/r/280772 (owner: 10Paladox) [09:30:49] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/280772 (owner: 10Paladox) [09:31:13] (03CR) 10jenkins-bot: [V: 04-1] [JsonData] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/280772 (owner: 10Paladox) [09:38:05] Yippee, build fixed! [09:38:06] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #841: 09FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/841/ [09:42:48] Yippee, build fixed! [09:42:48] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #787: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/787/ [11:14:02] 10Continuous-Integration-Config, 10Wikidata, 07Documentation, 13Patch-For-Review, 03Wikidata-Sprint-2016-04-12: [Story] autogenerated Wikibase documentation - https://phabricator.wikimedia.org/T133028#2217487 (10Lydia_Pintscher) p:05Triage>03Low [11:41:53] PROBLEM - Puppet run on integration-slave-trusty-1015 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:42:23] PROBLEM - Puppet run on deployment-cxserver03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:55:36] (03CR) 10Aashaka: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [11:58:31] Yippee, build fixed! [11:58:31] Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #459: 09FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/459/ [12:02:24] RECOVERY - Puppet run on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:21:59] RECOVERY - Puppet run on integration-slave-trusty-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [12:29:38] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:32:40] (03CR) 10Addshore: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [13:06:56] Yippee, build fixed! [13:06:56] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #889: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/889/ [13:13:52] PROBLEM - Puppet run on integration-slave-trusty-1015 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:15:26] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 15User-zeljkofilipin: Add helper to mediawiki_selenium for detecting if ResourceLoader module (JavaScript) has loaded - https://phabricator.wikimedia.org/T132753#2227145 (10zeljkofilipin) [13:24:23] (03CR) 10Aashaka: "thanks!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/279615 (https://phabricator.wikimedia.org/T116561) (owner: 10Aashaka) [13:45:12] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: mwext-testextension-hhvm-composer build timed out (after 30 minutes) - https://phabricator.wikimedia.org/T132681#2227242 (10Lydia_Pintscher) p:05Triage>03Normal [13:53:52] RECOVERY - Puppet run on integration-slave-trusty-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [13:57:32] (03CR) 10JanZerebecki: "No we should not forget about it, it is important to know when it is broken." [integration/config] - 10https://gerrit.wikimedia.org/r/284490 (owner: 10Paladox) [14:27:34] Yippee, build fixed! [14:27:35] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #860: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/860/ [14:30:29] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2227369 (10Osnard) Guys, thanks for your support on this. We are tring to shape BlueSpiceFoundation and BlueSpiceExtensions codebase to better integrate into the... [14:47:14] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:16] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 07Upstream, 15User-zeljkofilipin: [Bug] fix negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#2227413 (10zeljkofilipin) [15:40:37] 05Gerrit-Migration, 03releng-201617-q1, 10Differential, 10Continuous-Integration-Infrastructure (phase-out-gallium): Build glue from Differential to Gearman to Nodepool - https://phabricator.wikimedia.org/T130950#2227501 (10mmodell) [15:51:34] 10releng-201516-q3, 10Continuous-Integration-Infrastructure (phase-out-gallium), 07Jenkins: [keyresult] Migrate Jenkins to Jessie (gallium -> cobalt) - https://phabricator.wikimedia.org/T124121#2227539 (10hashar) [15:52:24] 05Continuous-Integration-Scaling, 06Operations, 10hardware-requests, 10Continuous-Integration-Infrastructure (phase-out-gallium): eqiad: (1) allocate server to migrate Zuul server to - https://phabricator.wikimedia.org/T95760#2227546 (10hashar) [15:55:41] 05Continuous-Integration-Scaling: Write a big picture overview about what the CI isolation project is and a rough roadmap - https://phabricator.wikimedia.org/T102137#2227557 (10hashar) 05Open>03Resolved After some thinking, the architecture documentation I wrote a year ago is a good enough big picture of the... [15:57:05] 05Continuous-Integration-Scaling: Write a big picture overview about what the CI isolation project is and a rough roadmap - https://phabricator.wikimedia.org/T102137#2227574 (10hashar) Overview of the target architecture will be produced via T133300 [16:25:28] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2227703 (10csteipp) I don't really like the direction, and would rather see us move towards a mo... [16:33:29] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2227746 (10csteipp) I should add, I think we can prevent having the agent's memory dumped (so I... [16:59:16] (03PS1) 10Paladox: [operations/debs/gerrit] Add gate and submit test [integration/config] - 10https://gerrit.wikimedia.org/r/284727 [17:14:08] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:58:59] (03CR) 10Hashar: [C: 04-1] "The debian-glue jobs are broken and the Gerrit one is falling / non voting. If the reason is solely to have Jenkins V+2 and merge as a co" [integration/config] - 10https://gerrit.wikimedia.org/r/284727 (owner: 10Paladox) [17:59:31] (03CR) 10Chad: [C: 04-1] "There's no need." [integration/config] - 10https://gerrit.wikimedia.org/r/284727 (owner: 10Paladox) [17:59:33] 10Beta-Cluster-Infrastructure, 06Labs, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2228059 (10Krenair) this just happened twice more :/ [18:02:50] !log disabling puppet on scap targets to test scap_3.1.0-1+0~20160421173204.70~1.gbp6706e0_all.deb [18:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:04:06] 10Beta-Cluster-Infrastructure, 06Labs, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2228070 (10hashar) Le 21/04/2016 19:59, Krenair a écrit : > this just happened twice more :/... [18:06:03] Project beta-scap-eqiad build #99288: 04FAILURE in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99288/ [18:08:26] hmm, somehow missed installing new version on deployment-jobrunner01 [18:10:13] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2228112 (10mmodell) @csteipp: Thanks for checking my assumptions. >>! In T133211#2227703, @cste... [18:15:59] Yippee, build fixed! [18:16:00] Project beta-scap-eqiad build #99289: 09FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99289/ [18:16:14] \o/ works. [18:38:14] 10Beta-Cluster-Infrastructure, 06Labs, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2228226 (10Krenair) cache-text04 again, I'll look into it [18:53:14] (03Abandoned) 10Paladox: [operations/debs/gerrit] Add gate and submit test [integration/config] - 10https://gerrit.wikimedia.org/r/284727 (owner: 10Paladox) [19:04:55] RECOVERY - Host integration-trusty-1026 is UP: PING OK - Packet loss = 0%, RTA = 1.05 ms [19:07:50] !log scap version testing should be done, puppet should no longer be disabled on hosts [19:07:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:09:37] 03Scap3, 10scap: [scap] Consolidate scripts as sub-commands of `scap` - https://phabricator.wikimedia.org/T67827#2228331 (10thcipriani) 05Open>03Resolved [19:31:14] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [19:38:27] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2225168 (10hashar) I am not a fan of keyholder and multiplying deploy users for each software we... [19:51:10] (03PS1) 10Smalyshev: Enable tests for kafka-watcher [integration/config] - 10https://gerrit.wikimedia.org/r/284746 [20:13:02] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [20:30:18] twentyafterfour, would you be the appropriate person to talk to about scap3? [20:31:59] oh no, sorry [20:32:18] turns out this thing uses trebuchet [20:53:58] Krenair: yeah, I know scap3 pretty well, trebuchet not so much (although I know it a little) [20:54:10] (03PS5) 10Awight: Experimental lint and phpunit for the SmashPig repo [integration/config] - 10https://gerrit.wikimedia.org/r/284618 (https://phabricator.wikimedia.org/T133248) [20:54:37] Krenair: I might be able to help with trebuchet [20:55:09] (03Abandoned) 10Awight: Experimental lint and phpunit for the SmashPig repo [integration/config] - 10https://gerrit.wikimedia.org/r/284618 (https://phabricator.wikimedia.org/T133248) (owner: 10Awight) [20:55:15] although when it goes really wrong I get lost. salt pillars and grains still don't make much sense to me [20:55:45] getting lost with salt first, see the other labs channel [20:56:42] 10Continuous-Integration-Config, 10Fundraising-Backlog, 07FR-ActiveMQ, 07FR-Smashpig, and 2 others: SmashPig CI should run phpunit - https://phabricator.wikimedia.org/T133248#2228713 (10awight) 05Open>03Invalid It was already working! [21:27:39] Hi im wondering if wikimedia will upgrade to use Ubuntu 16.04 lts instead of debian for the reason it now includes support for snap packages [21:28:04] Which allow quicker creation of packages like windows and it is secure and allows you to update on the fly [21:28:21] Without needing to update os from 16.04 to another one to get new features [21:35:09] twentyafterfour, bd808: wait what, role::deployment::server includes both trebuchet and scap stuff? [21:35:20] yeah :/ [21:35:41] I've wanted to split those apart for years now [21:35:59] what's stopping it? [21:36:09] typing and testing [21:37:03] there's a trebuchet section to this file... I'm gonna try moving it out and including it instead [21:37:15] Krenair: awesome! give it a shot [21:37:34] the scap bits should be fairly self contained [21:37:56] the trebuchet side may sprawl all over, I'm not sure [21:43:03] 10Continuous-Integration-Infrastructure, 07Technical-Debt, 07Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#2229094 (10Paladox) Only one left is https://phabricator.wikimedia.org/T63607 I'm not sure weather to close it since the extension is not activ... [21:44:10] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2229100 (10mobrovac) I agree with @csteipp that a general solution is needed, but I also think t... [21:50:13] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2229106 (10Paladox) Your welcome. [21:51:02] paladox: I doubt it. You'd have to talk to #wikimedia-operations about distros but we're migrating from ubuntu to debian so I doubt it'd be easy to convince them that we need snap packages. [21:54:04] Ok thanks for replying. [21:54:09] bd808, well... puppet runs... [21:56:04] Krenair: that's a start! [21:56:31] splitting the roles now will make killing trebuchet entirely later easier [21:57:02] Error: Execution of '/usr/bin/salt-call --log-level=quiet --out=json deploy.fetch servermon/servermon' returned 1: Traceback (most recent call last): [21:57:10] redis.exceptions.ResponseError: READONLY You can't write against a read only slave. [21:58:25] seems $::fqdn is different to hiera('deployment_server', 'tin.eqiad.wmnet') in my setup [21:58:49] oh because I didn't set it to the actual fqdn, oops :) [22:03:08] bd808, so I should have a directory called "/srv/deploy/" on the deployment server? [22:03:11] not just /srv/deployment/ ? [22:04:07] I don't see a /srv/deploy on deployment-tin [22:08:53] maybe just a doc issue [22:11:09] hm [22:11:27] role::deployment::repo_config includes a servermon/servermon key [22:11:46] so theoretically /srv/deployment/servermon/servermon should exist [22:12:04] but it doesn't [22:16:09] on the deployment server? might see if it's bailing out trying to do the deploy-server init thing: sudo salt-call deploy.deployment_server_init [22:20:05] Project beta-update-databases-eqiad build #8027: 04FAILURE in 5.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8027/ [23:01:55] thcipriani, Function deploy.deployment_server_init is not available [23:03:50] hmm, my notes mention that the deploy.py file with the deployment_server_init function should be under: /var/cache/salt/minion/files/base/_modules/ [23:04:19] if it's not there, you might need to check the salt master for why that would be. May just needs a sync from there? [23:06:54] the deployment server is the salt master, and also the puppet master [23:07:42] /var/cache/salt/minion/files/base does not contain a _modules directory [23:08:49] root@sm-puppetmaster-trusty2:~# cat /var/cache/salt/master/file_lists/roots/base.p [23:08:49] ��files��top.sls�deploy/sync_all.sls�_modules/deploy.py�_returners/deploy_redis.py�dirs��.�deploy�_modules�_returners�links��empty_dirs� [23:11:58] I know that the deployment_server class calls: salt '*' saltutil.sync_all but if the deploy module isn't on the minion, maybe that failed somewhere along the line. [23:12:51] there is, evidently, also a sync_modules https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.sync_modules [23:14:26] aha [23:14:43] I ran sudo salt '*' saltutil.sync_all [23:14:47] root@sm-puppetmaster-trusty2:~# ls /var/cache/salt/minion/files/base/_modules [23:14:47] deploy.py [23:15:43] nice, so now all the sudo salt-call deploy.[whatever] functions *should* work, they may fail for other equally esoteric reasons, but that's something :) [23:15:55] I put class { 'deployment::deployment_server' in the wrong place [23:20:26] Yippee, build fixed! [23:20:27] Project beta-update-databases-eqiad build #8028: 09FIXED in 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8028/ [23:53:22] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 07Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#2229564 (10Riley_Huntley) https://meta.wikimedia.org/wiki/Special:CentralAuth/Momonyc Exception encountered, of type "Exception"