[00:40:22] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-sca01.puppetagent.failed_events.value (33.33%) [00:59:58] RECOVERY - BetaLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [03:19:07] (03PS2) 10Krinkle: Assertion macros for node js version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [03:30:48] (03PS3) 10Krinkle: Add macros for asserting node.js version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [03:30:50] (03CR) 10Krinkle: Add macros for asserting node.js version (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [03:31:37] (03CR) 10Krinkle: Add macros for asserting node.js version (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [03:31:47] (03CR) 10Krinkle: "Adjusted indentation and made API/output less verbose." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [03:44:20] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #165: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/165/ [05:17:50] Yippee, build fixed! [05:17:50] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #198: FIXED in 6 min 52 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/198/ [05:32:40] Yippee, build fixed! [05:32:40] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #215: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/215/ [05:59:15] Yippee, build fixed! [05:59:15] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #221: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/221/ [07:04:32] Yippee, build fixed! [07:04:32] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #226: FIXED in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/226/ [08:02:57] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #168: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/168/ [08:04:36] (03CR) 10Zfilipin: [C: 032] Resolution of Sauce session ID with custom browser [selenium] - 10https://gerrit.wikimedia.org/r/161627 (owner: 10Dduvall) [08:04:52] (03Merged) 10jenkins-bot: Resolution of Sauce session ID with custom browser [selenium] - 10https://gerrit.wikimedia.org/r/161627 (owner: 10Dduvall) [08:05:13] (03PS9) 10Zfilipin: Bundle macro + ruby doc using yard [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [08:09:17] (03CR) 10Zfilipin: [C: 031] "Looks good to me. Antoine, would you like to merge this? I did not +2 it because of this comment: "leaving this change open so we can twea" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [08:14:07] (03CR) 10Hashar: "Zeljkof, I am holding this change till I find some time to process Dan comment above. We have Ubuntu Trusty slaves, so we can probably mo" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [08:38:32] hashar: oops, forgot to start my irc client this morning :) [08:38:39] zeljkof: shit happens :D [08:38:55] zeljkof: I will probably get the job templates that uses bundle / gem to be run on Trusty instance [08:39:01] so we effectively switch to ruby2 [08:39:11] but I have probably have little bandwidth this week to do so [08:39:30] should all be about making sure ruby2 is installed on Trusty instances (such as integration-slave1008.eqiad.wmflabs ) [08:39:42] and then have the bundle job template to use node: UbuntuTrusty :D [08:39:57] hashar: great [08:40:47] zeljkof: the bundle job template will probably greatly simplify our templates :-] [08:41:19] hashar: yes, looks like it removes some duplication [08:43:07] gotta fix a zuul bug first :] [08:43:23] 3Wikimedia / 3Continuous integration: Zuul cloner: fails on extension jobs against a wmf branch - 10https://bugzilla.wikimedia.org/71133#c1 (10Antoine "hashar" Musso) 5NEW>3ASSI p:5Unprio>3Highes a:3Antoine "hashar" Musso I am 100% focusing on this issue which is a bug/lack of feature in Zuul [08:52:52] (03PS2) 10Zfilipin: Stricter pending behavior for falsely passing steps [selenium] - 10https://gerrit.wikimedia.org/r/160024 (https://bugzilla.wikimedia.org/56243) (owner: 10Dduvall) [08:53:40] (03CR) 10Zfilipin: [C: 032] Stricter pending behavior for falsely passing steps [selenium] - 10https://gerrit.wikimedia.org/r/160024 (https://bugzilla.wikimedia.org/56243) (owner: 10Dduvall) [08:53:54] (03Merged) 10jenkins-bot: Stricter pending behavior for falsely passing steps [selenium] - 10https://gerrit.wikimedia.org/r/160024 (https://bugzilla.wikimedia.org/56243) (owner: 10Dduvall) [09:25:08] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-sca01.puppetagent.failed_events.value (33.33%) [12:28:31] (03PS1) 10Zfilipin: Added a few more languages to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162224 [12:30:14] (03PS2) 10Zfilipin: Add a few more languages to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162224 [12:31:00] (03CR) 10Amire80: [C: 032] Add a few more languages to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162224 (owner: 10Zfilipin) [12:39:02] aharoni: https://www.mediawiki.org/wiki/Amsterdam_Hackathon_2014 [12:44:07] Project browsertests-VisualEditor-language-screenshot-linux-firefox-sauce » yue,contintLabsSlave && UbuntuPrecise build #94: ABORTED in 4 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox-sauce/LANGUAGE_SCREENSHOT_CODE=yue,label=contintLabsSlave%20&&%20UbuntuPrecise/94/ [12:44:24] Project browsertests-VisualEditor-language-screenshot-linux-firefox-sauce » gl,contintLabsSlave && UbuntuPrecise build #94: ABORTED in 4 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox-sauce/LANGUAGE_SCREENSHOT_CODE=gl,label=contintLabsSlave%20&&%20UbuntuPrecise/94/ [12:47:32] (03PS3) 10Zfilipin: WIP Running language screenshot job using local Firefox [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/154052 [12:49:34] (03Abandoned) 10Zfilipin: WIP Running language screenshot job using local Firefox [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/154052 (owner: 10Zfilipin) [12:55:12] (03PS1) 10Zfilipin: WIP Running language screenshot job using local Firefox [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162225 (https://bugzilla.wikimedia.org/69535) [12:55:48] (03CR) 10Zfilipin: "Moved to https://gerrit.wikimedia.org/r/#/c/162225/" [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/154052 (owner: 10Zfilipin) [13:09:54] Project browsertests-VisualEditor-language-screenshot-linux-firefox » yue,contintLabsSlave && UbuntuPrecise build #3: FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=yue,label=contintLabsSlave%20&&%20UbuntuPrecise/3/ [13:22:55] RECOVERY - BetaLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [13:35:18] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-sca01.puppetagent.failed_events.value (33.33%) [13:59:40] RECOVERY - BetaLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [14:03:38] 3Wikimedia / 3Continuous integration: Zuul cloner: fails on extension jobs against a wmf branch - 10https://bugzilla.wikimedia.org/71133#c2 (10Antoine "hashar" Musso) Patch proposed upstream: https://review.openstack.org/#/c/123437/ Will let Zuul cloner clone a repo whenever /.git/ does not exist. [14:22:26] (03CR) 10Hashar: "So do we want to generate screenshots from our instances or using SauceLabs? :)" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162225 (https://bugzilla.wikimedia.org/69535) (owner: 10Zfilipin) [14:30:51] (03CR) 10Zfilipin: [C: 031] "Looks good to me. I have left a couple of nitpick comments." (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:31:35] hashar: re https://gerrit.wikimedia.org/r/#/c/162225/ [14:32:00] looks like sauce labs does not have chinese and japanese fonts installed on the machines [14:32:33] so we are trying running the screenshot job from a local firefox, where according to this bug, fonts should be installed https://bugzilla.wikimedia.org/show_bug.cgi?id=69535 [14:32:47] but the screenshots say the fonts are not installed [14:33:34] hashar: see any screenshot here [14:33:35] https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=yue,label=contintLabsSlave%20&&%20UbuntuPrecise/ws/log/ [14:34:17] will try to create a small script to reproduce the problem [14:51:34] (03PS4) 10Hashar: Add macros for asserting node.js version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 [14:53:44] (03CR) 10Hashar: "Done, with PS4, the version parameter is passed directly to grep which supports basic regular expression. That gives more freeform." (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161918 (owner: 10Hashar) [14:53:54] (03PS6) 10Hashar: parsoidsvc: split npm job based on nodejs version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160589 (owner: 10Jforrester) [14:54:00] off [14:54:04] be back for the QA checkin [14:59:59] (03CR) 10jenkins-bot: [V: 04-1] parsoidsvc: split npm job based on nodejs version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160589 (owner: 10Jforrester) [15:24:50] zeljkof: hi [15:25:09] aharoni: hi [15:26:51] zeljkof: I reopened https://bugzilla.wikimedia.org/show_bug.cgi?id=69535 , for what it's worth [15:26:52] 3Wikimedia / 3Quality Assurance: Fonts for Japanese and Chinese must be installed for VisualEditor localized screenshots - 10https://bugzilla.wikimedia.org/69535#c8 (10Amir E. Aharoni) 5RESO/WOR>3REOP The jobs as they are configured now run on Sauce, whee we cannot control the fonts. We need to run the... [15:27:16] can you please add relevant information there about what we tried to do today? [15:27:23] aharoni: sorry, got distracted, will add more data to the bug (it is on my list) [15:27:28] Thanks. [15:34:25] aharoni: will report it to sauce too [15:34:37] they might be able to fix the problem on their end too [15:54:47] lets try joining hangout [16:03:20] Reedy: meeting ping [16:03:29] yeah, i just realised [16:03:41] just had to go on a poop hunt [16:05:12] ... tmi [18:08:39] 3Wikimedia / 3Quality Assurance: Browser tests should not use Watir API - 10https://bugzilla.wikimedia.org/70287#c1 (10Chris McMahon) In some cases, the watir API is required. The most important case is when it is necessary to identify an element by more than one locator, which is not supported in selenium... [18:25:20] marxarelli: well ranted. :) [18:26:09] greg-g: i still got it! [18:26:34] (cut to montage) [18:26:56] urgh, I've been defending my choice of browser test tools for almost 3 years. anything to turn focus off the fact that we suck at testing in general. [18:30:28] heh [18:30:29] i hope i didn't hate on cucumber too much. like i said (tried to say anyway) the feature text is really valuable but it has to serve the user-designer-developer contract [18:33:22] just as an exercise, think about how a feature would read if it were integrated into an rfc... [19:02:12] hello [19:09:11] greg-g: do you have an example of a "what i did" page? [19:09:56] marxarelli: not off the top of my head, no :/ [19:11:51] greg-g: cool, np. i'll wing it [19:12:49] greg-g: but a wiki page is probably best, you think? [19:13:09] yeah [19:36:54] (03PS1) 10Dduvall: Releasing minor version 0.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/162366 [19:38:25] chrismcmalunch: when you get a chance ^ [19:39:23] the mmv perf test needs that custom browser fix :/ [19:52:24] 3Wikimedia Labs / 3deployment-prep (beta): deployment-graphite.eqiad.wmflabs went away? - 10https://bugzilla.wikimedia.org/71031#c3 (10C. Scott Ananian) Ok, I've changed the statsd configuration from deployment-graphite.eqiad.wmflabs to labmon1001.eqiad.wmnet on both deployment-pdf01 and deployment-pdf02. F... [19:58:06] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c1 (10Greg Grossmeier) p:5Unprio>3High Reedy: can you take a look here? There's also https://bugzilla.wikimedia.org/show_bug.cgi?id=70145 which is "Safari sets forceH... [19:58:21] 3Wikimedia Labs / 3deployment-prep (beta): Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - 10https://bugzilla.wikimedia.org/70145 (10Greg Grossmeier) [19:59:55] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079 (10Greg Grossmeier) [19:59:55] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c2 (10Greg Grossmeier) And https://bugzilla.wikimedia.org/show_bug.cgi?id=70948 "Beta Cluster isn't redirecting en.wikipedia.beta.wmflabs.org correctly" but I thought tha... [19:59:55] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster isn't redirecting en.wikipedia.beta.wmflabs.org correctly - 10https://bugzilla.wikimedia.org/70948 (10Greg Grossmeier) [20:11:09] 3Wikimedia / 3Continuous integration: Zuul cloner: fails on extension jobs against a wmf branch - 10https://bugzilla.wikimedia.org/71133#c3 (10Antoine "hashar" Musso) The patch I wrote fails with python2.6. I have no idea whether it is py2.6 only issue (GitPython could have a different behavior) or if it hi... [20:11:38] and sleeping now. yeah really. [20:18:00] (03CR) 10Cmcmahon: [C: 032] Releasing minor version 0.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/162366 (owner: 10Dduvall) [20:18:16] (03Merged) 10jenkins-bot: Releasing minor version 0.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/162366 (owner: 10Dduvall) [20:18:24] marxarelli|lunch: mergededed [20:20:18] chrismcmahon: right on! thanks! [20:21:38] marxarelli: if you have a few minutes, could you look over https://gerrit.wikimedia.org/r/#/c/162144/ ? that's the one that has the @custom_browser fix and also puts Actual Assertions in the Then step o_O [20:22:01] I'mma tryna fix up this repo pretty [20:22:08] 3Wikimedia Labs / 3deployment-prep (beta): Determine first pass list of icinga-alerting data from graphite.wmflabs - 10https://bugzilla.wikimedia.org/70141#c17 (10Greg Grossmeier) 5PATC>3ASSI p:5High>3Normal a:3Yuvi Panda Yuvi: Thanks for the first pass work! Once you remove yourself from the list... [20:22:09] chrismcmahon: for sure. i'll take a look [20:23:28] chrismcmahon: actually, you may want to update the gemfile.lock with the new mw-selenium release. it includes a fix for custom browsers and sauce [20:27:23] 3Wikimedia Labs / 3deployment-prep (beta): deployment-graphite.eqiad.wmflabs went away? - 10https://bugzilla.wikimedia.org/71031#c4 (10C. Scott Ananian) 5NEW>3RESO/FIX Seems like it's working now. [20:27:40] marxarelli: good idea, thanks, one moment... [20:29:36] marxarelli: that should do it... [20:30:28] chrismcmahon: i'm leaving some nit-picky ruby stuff which is just a suggestion seeing as it's not your implementation, you just moved it [20:31:22] marxarelli: np, I will pick the nits [20:36:40] marxarelli: that should do it [20:41:14] chrismcmahon: cool, it's green for me. i can +2 it if you're done [20:42:50] marxarelli: yes, please, I'd like to build on that commit without having rebase hell [20:45:08] 3Wikimedia / 3Quality Assurance: Cucumber step should fail if pending RSpec expectation no longer fails - 10https://bugzilla.wikimedia.org/56243#c19 (10Dan Duvall) 5PATC>3RESO/FIX Released in 0.4.0. [20:46:28] and exemplary update :-) [20:54:10] the new RSpec syntax will take some getting used to. I guessed at "to_not" but it was "not_to". [20:55:13] and that didn't actually work :( [20:57:07] not_to sounds more natural but i have no idea which is grammatically correct :) [20:57:46] i'll ask my wife. she's the writer [21:01:02] and yes folks, parentheses are important! [21:07:46] (03CR) 10Dduvall: "Cool, thanks for the feedback! I'll keep working on this right after I wrap up the MMV perf stuff." (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [22:04:40] CUSTOM - BetaLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.puppetagent.time_since_last_run.value (100.00%) deployment-prep.deployment-mediawiki03.puppetagent.time_since_last_run.value (100.00%) deployment-prep.deployment-mediawiki01.puppetagent.time_since_last_run.value (100.00%) [22:17:54] man, search is dragging on beta again. I wonder if we have a real issue [22:30:18] chrismcmahon: worth calling in nik/chad to take a look right now? [22:31:30] greg-g: nah, I asked Nik about it earlier, he'll probably think it's just an artifact of a constrained system. but we seem to be bumping the constraints a lot in just the past couple of weeks, and that often points to some sort of issue. [22:32:28] greg-g: there is a MobileFrontend test that is sort of a canary for Search performance. [22:39:55] chrismcmahon: they don't look terribly overloaded: http://people.wikimedia.org/~gjg/betacluster/graphs.html (my quick/hacky dashboard that isn't pretty) [22:40:03] chrismcmahon: the search ones are last day, the rest are last week [22:44:17] I'm not sure Jenkins time stamps are believable, but that little spike around noon likely corresponds to the failure at https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/lastCompletedBuild/testReport/ [22:44:36] why wouldn't they be believable? [22:46:26] greg-g: yep, email hit my inbox at 2:45 PM, so that test ran around 2-2:30, right at the middle of the search load. [22:48:34] 2:45 mountain? [22:49:09] looks like the CPU spike was at about 13:00 UTC [22:50:16] greg-g: 2:45 Pacific, which is the same as Arizona (until the next time change) [22:51:05] chrismcmahon: gah, right, stupid DST oddities :) [22:51:52] !log Jenkins stuck trying to update database in beta again with the dumb "waiting for executors" bug/problem [22:51:54] Logged the message, Master [22:53:23] !log The dumb "waiting for executors" bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=70597 [22:53:24] Logged the message, Master [22:55:38] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c7 (10Greg Grossmeier) Sept 23rd: 22:51 bd808: Jenkins stuck trying to update database in beta again with the dumb "waiting for executors" bug/pro... [23:03:28] The stop/start dance isn't helping this time :( [23:06:07] brb, sacrificing a goat [23:06:36] chrismcmahon: That helped :) [23:08:00] !log Jenkins and deployment-bastion talking to each other again after six (6!) disconnect, cancel jobs, reconnect cycles [23:08:02] Logged the message, Master [23:08:41] I think maybe the trick involves disconnect, wait... wait... wait..., reconnect [23:17:54] 3Wikimedia / 3Continuous integration: Jenkins: Detect text before the first 3None Krinkle: Over a year since last activity, removing Krinkle as assignee. He know how to re-add if he wants :) [23:18:23] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c8 (10Bryan Davis) (In reply to Bryan Davis from comment #6) > > I have manually changed the config for the beta-update-databases-eqiad job > in... [23:18:56] 3Wikimedia / 3Continuous integration: Update doxygen from 1.7.x to 1.8.x on gallium - 10https://bugzilla.wikimedia.org/46771#c15 (10Greg Grossmeier) a:5Krinkle>3None Over a year since Krinkle's last activity, removing as assignee. He knows how to re-add if he wants :) [23:28:36] PROBLEM - BetaLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.puppetagent.time_since_last_run.value (100.00%) deployment-prep.deployment-mediawiki03.puppetagent.time_since_last_run.value (100.00%) deployment-prep.deployment-mediawiki01.puppetagent.time_since_last_run.value (100.00%)