[08:17:09] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1298563 (10BBlack) p:5Triage>3Normal [08:17:27] 10Beta-Cluster, 10Deployment-Systems, 10Traffic, 6operations: Upgrade beta-cluster caches to jessie - https://phabricator.wikimedia.org/T98758#1298566 (10BBlack) p:5Triage>3Normal [08:56:42] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: Connection refused [09:03:17] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 100.00% of data above the critical threshold [0.0] [09:22:24] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [10:28:58] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298692 (10ArielGlenn) 3NEW a:3ArielGlenn [10:29:15] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298700 (10ArielGlenn) p:5Triage>3Normal [10:29:19] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298692 (10ArielGlenn) Looks similar: https://github.com/saltstack/salt/issues/23047 [10:30:12] anyone here that can help with karma + qunit + jenkins? [10:30:21] https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9784/console [10:39:14] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298707 (10ArielGlenn) 2015-05-20 10:14:40,401 [salt.log.setup ][ERROR ] An un-handled exception was caught by salt's global exception handler: TypeError: unhashable type... [10:40:43] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298709 (10ArielGlenn) more log: 2015-05-20 10:14:40,398 [salt.utils.minions][ERROR ] Failed matching available minions with list pattern: [{'mw1190.eqiad.wmnet': {'ret':... [12:14:34] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1298794 (10Anomie) >>! In T99411#1298146, @mmodell wrote: > @anomie: I'm not sure I understand the last thing about multiple changes - the changes would be in separate commits just like they are in other situ... [12:19:21] etonkovidova: can you please share your doc with articles to check ? [12:42:29] 10Deployment-Systems, 6operations: Unhashable type: dict error when running salt --batch-size - https://phabricator.wikimedia.org/T99776#1298827 (10ArielGlenn) I have tested a manual fix in deployment-prep which seems ok. I have commented with this fix and the explanation on the upstream bug and will be naggi... [13:10:20] Yippee, build fixed! [13:10:21] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #652: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/652/ [14:21:55] 10Deployment-Systems: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1298972 (10ArielGlenn) The salt master only knows about minions because it has their public keys. If the key has not been deleted on the master, then as far as the salt master is concerned, t... [14:32:45] 10Beta-Cluster, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver: Beta: cxserver fails to start with permission issue - https://phabricator.wikimedia.org/T98436#1298982 (10KartikMistry) 5Open>3declined a:3KartikMistry This seems temporary issue. [14:37:58] Yippee, build fixed! [14:37:58] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #125: FIXED in 9 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/125/ [14:47:04] Yippee, build fixed! [14:47:04] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #258: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/258/ [14:49:43] 10Deployment-Systems: Trebuchet doesn't like when a deployer server is also a minion, a edge case for scap - https://phabricator.wikimedia.org/T67549#1299025 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/212291/ this? [14:51:47] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<20.00%) [14:58:38] * AndyRussG waves at releng [14:59:02] Hi, suggestions for merging and deploying this and the related patches (mentioned in the commit message) https://gerrit.wikimedia.org/r/#/c/202925/ ? [15:29:05] Yippee, build fixed! [15:29:05] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #539: FIXED in 1 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/539/ [15:34:46] zeljkof: qunit issues with wikidata [15:36:29] 6Release-Engineering, 6Phabricator: Next Phabricator upgrade on 2015-05-27 (tentative) - https://phabricator.wikimedia.org/T98451#1299119 (10Aklapper) [15:41:55] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1299140 (10mmodell) @anomie: Regarding swat, I didn't intend to suggest fundamentally changing the way swat works, you wouldn't have to deploy the patches all at once, you could do them one at a time. I'm sim... [15:45:18] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1299144 (10phuedx) 3NEW [15:45:27] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1299151 (10phuedx) [15:45:58] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1299144 (10phuedx) [15:59:04] !log Applied role::beta::puppetmaster on deployment-salt to get Puppet logstash reports back [15:59:21] Logged the message, Master [16:00:01] Some nice people cleaned up my toys when they got rid of arbitrary prod<->beta differences [16:04:19] 10Deployment-Systems, 5Patch-For-Review: Trebuchet can mess up .git permissions on deployment server - https://phabricator.wikimedia.org/T94754#1299186 (10bd808) [16:12:30] 10Deployment-Systems: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1299206 (10bd808) Making it automatic might be hard, but adding a `git deploy targets --remove ` command to automate the redis srem would at least make manual cleanups a little less... [16:12:44] hey zeljkof! can you help us with an issue we aer having with qunit and jenkins? [16:12:55] it'S kind of strange, when running these tests locally, they work: https://phabricator.wikimedia.org/T99775 [16:13:17] you are kind of like my last hope, everyone who knows about this stuff seems to be traveling... [16:17:03] DanielK_WMDE_: I'm pretty sure he's at an off-site in France right now [16:19:15] but that doesn't start until saturday :) [16:19:25] or is the wmf having a special pre-hackathon thingy there? [16:19:32] 10Beta-Cluster, 6Release-Engineering: Determine weekly triage meeting for Beta Cluster - https://phabricator.wikimedia.org/T98204#1299208 (10Aklapper) Whenever this might take place, please consider adding me to the invitation list as tentative. TIA [16:19:33] anyway - i'm looking forward to meeting him there. [16:19:57] but who can fix the jenkins breakage now?... [16:20:22] it seems like krinkle, addshore, janz, and hashar are all out :( [16:21:09] DanielK_WMDE_: the releng team is having an offsite before lyon [16:21:17] 10Deployment-Systems, 6Release-Engineering: Determine weekly triage meeting for Deployment Systems - https://phabricator.wikimedia.org/T98206#1299210 (10Aklapper) Whenever this might take place, please consider adding me to the invitation list as tentative. TIA [16:21:34] <^demon|zzz> DanielK_WMDE_: I can find him and ping him perhaps, he's around the hotel somewhere. [16:22:18] legoktm: ah, nice. [16:23:05] ^demon|zzz: thanks! the perfect person would be Krinkle, I think he set this up. But zeljkof or hashar may be able to help. [16:23:22] <^demon|zzz> I'll see if I can snag one of them and tell them to find you here. [16:24:08] ^demon|zzz: just point them at https://phabricator.wikimedia.org/T99775 in case i'm offline [16:24:14] <^demon|zzz> Will do [16:24:17] * ^demon|zzz notes task # [16:24:26] thanks a ton! [16:24:32] DanielK_WMDE_: I am also on the road :) [16:24:40] Will take a look [16:24:58] I'll be off for a couple of hourse soon, but will be online around 11pm CEST [16:25:13] <^demon|zzz> Oh there's zeljkof [16:25:18] * ^demon|zzz was magic [16:25:20] <^demon|zzz> Clearly! [16:25:25] zeljkof: oh hey :) looking forward to see you all in france! [16:25:51] zeljkof, ^demon|zzz: still if you find Krinle, poke him please. he probably knows best what the problem is [16:26:14] <^demon|zzz> I dunno where Krinkle is, not with us :p [16:26:18] <^demon|zzz> We be in Annecy, not Lyon [16:26:31] ah, too bad [16:26:54] * zeljkof is looking for laptop and beer [16:26:57] probably try to just delete our caches (of mediawiki) [16:27:03] i think we try to reuse the last clone [16:27:06] <^demon|zzz> DanielK_WMDE_: vue sur le lac: https://instagram.com/p/2zEc9hG0a1 :) [16:27:13] maybe cleaning it will work [16:27:37] ooooh, pretty! [16:27:53] https://gerrit.wikimedia.org/r/#/c/212284/ patch on WikibaseJavaScriptApi did not have this issue but runs the same tests [16:29:06] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:41:21] hashar, thcipriani|afk and me are looking at https://phabricator.wikimedia.org/T99775 [16:48:18] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [16:55:19] 6Release-Engineering, 6Phabricator: Next Phabricator upgrade on 2015-05-27 (tentative) - https://phabricator.wikimedia.org/T98451#1299311 (10Aklapper) [16:57:08] 6Release-Engineering, 6Phabricator: Next Phabricator upgrade on 2015-05-27 (tentative) - https://phabricator.wikimedia.org/T98451#1268189 (10Aklapper) [16:57:22] 6Release-Engineering, 6Phabricator: Next Phabricator upgrade on 2015-05-27 (tentative) - https://phabricator.wikimedia.org/T98451#1299327 (10Aklapper) [16:57:37] 6Release-Engineering, 6Phabricator: Next Phabricator upgrade on 2015-05-27 (tentative) - https://phabricator.wikimedia.org/T98451#1268189 (10Aklapper) [16:58:18] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.017 second response time [17:18:59] !log add --fail to curl inside mwext-Wikibase-qunit jenkins job [17:19:03] Logged the message, Master [17:21:56] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 6operations, 5Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1299420 (10faidon) [17:36:43] 10Beta-Cluster, 6Release-Engineering: Determine weekly triage meeting for Beta Cluster - https://phabricator.wikimedia.org/T98204#1299459 (10hashar) @mmodell scheduled it in our calendars on Mondays at 18:50 – 19:20 UTC. [17:46:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1299480 (10hashar) It might or not be an issue with CI infra. @thcipriani @zeljkofilipin and I are taking this as an opportunity... [17:47:37] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1299489 (10zeljkofilipin) A tweak that did not help. https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/jobConfigHis... [18:00:02] twentyafterfour: You might be interested in this toy I started working on last night -- https://gerrit.wikimedia.org/r/#/c/212294/ -- Vagrantfile for ops/puppet [18:01:10] bd808: very cool [18:01:43] Its pretty sketchy so far but at least a semi-working start [18:13:03] bd808: cool, I will try it out [18:21:31] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL 100.00% of data above the critical threshold [0.0] [18:21:54] ^demon|away: bd808 i need rights to push tags and branches to WikibaseJavaScriptApi on gerrit [18:22:14] or how do i tag something in the gerrit UI? [18:22:49] oh, looks like maybe i can grant this [19:26:40] (03PS1) 10Legoktm: Upgrade composer to 1.0.0-alpha10 [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 [19:27:07] (03PS2) 10Legoktm: Upgrade composer to 1.0.0-alpha10 [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 [19:32:11] (03PS1) 10Legoktm: Rebuild autoloader to support classmap authoritative. [integration/composer] - 10https://gerrit.wikimedia.org/r/212325 [19:33:57] 10Continuous-Integration-Infrastructure: Upgrade integration/composer to 1.0.0-alpha10 - https://phabricator.wikimedia.org/T99819#1299761 (10Legoktm) 3NEW a:3Legoktm [19:34:42] (03PS3) 10Legoktm: Upgrade composer to 1.0.0-alpha10 [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 (https://phabricator.wikimedia.org/T99819) [19:34:44] (03PS2) 10Legoktm: Rebuild autoloader to support classmap authoritative. [integration/composer] - 10https://gerrit.wikimedia.org/r/212325 [20:05:27] <^demon|away> aude: Did you get it figured out? [20:05:44] <^demon|away> (also, weird. those should be granted to repo owners on mw/* projects by default) [20:27:43] (03CR) 10BryanDavis: [C: 031] Upgrade composer to 1.0.0-alpha10 [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 (https://phabricator.wikimedia.org/T99819) (owner: 10Legoktm) [20:43:34] (03CR) 10TheDJ: "@hashar, should be fixed with I7ebfcc1da89147e24d5bee8b72628ea249a882cb" [integration/config] - 10https://gerrit.wikimedia.org/r/209991 (owner: 10TheDJ) [21:17:24] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL 100.00% of data above the critical threshold [43200.0] [21:45:28] ^demon|away: figured out the gerrit permissions [21:45:41] (not the jenkins - qunit issues) [22:39:50] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300322 (10hashar) The [[ https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/ | mwext-Wikibase-qunit ]] job started f... [22:41:29] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300325 (10hashar) a:5Krinkle>3None [22:43:00] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1298655 (10hashar) Removed @krinkle from assignee, doesn't make sense to force assign someone without pr... [22:46:51] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300343 (10daniel) > 1 buffered job(s) never inserted. I have seen this locally, screwing up API respon... [22:47:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300347 (10aude) @hashar but strange that even with pulling master core, I don't get the failures locall... [22:57:02] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300379 (10Legoktm) Related to T99849? [22:58:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, 7Jenkins: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300392 (10JanZerebecki) curl --fail only ommits any output in the HTTP status failure case. In sh false... [23:16:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, and 2 others: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300439 (10hashar) >>! In T99775#1300343, @daniel wrote: >> 1 buffered job(s) never inserted. > > I h... [23:22:39] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, and 2 others: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1298655 (10hashar) >>! In T99775#1300392, @JanZerebecki wrote: > Regarding the curl error: curl --fail...