[02:15:09] Project beta-scap-eqiad build #26277: FAILURE in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/26277/ [02:25:09] Yippee, build fixed! [02:25:10] Project beta-scap-eqiad build #26278: FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/26278/ [04:43:21] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #273: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/273/ [04:56:55] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #211: FAILURE in 1 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/211/ [05:30:27] Yippee, build fixed! [05:30:27] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #346: FIXED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/346/ [06:19:37] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #224: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/224/ [07:07:00] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #223: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/223/ [07:07:50] Any reason, I can't create m1.large instance in beta labs? (deployment-prep) [07:07:56] quota foo? [07:51:53] 3Wikimedia / 3Quality Assurance: review Cucumber performance experiment - 10https://bugzilla.wikimedia.org/63118#c1 (10Željko Filipin) Related commit is merged. Can this be resolved? [07:53:09] 3Wikimedia / 3Quality Assurance: Investigate how often browser tests fail because of Jenkins performance problems (tracking) - 10https://bugzilla.wikimedia.org/60338 (10Željko Filipin) a:3Željko Filipin [07:54:07] 3Wikimedia / 3Quality Assurance: Selenium bug for IE10 causes test failures for Flow - 10https://bugzilla.wikimedia.org/59801#c4 (10Željko Filipin) Is this resolved? [08:10:33] hmm. Where is deployment-salt puppet logs? [08:18:53] 3Wikimedia / 3Quality Assurance: Activate ChuckNorris Plugin after Jenkins builds - 10https://bugzilla.wikimedia.org/63305#c4 (10Antoine "hashar" Musso (WMF)) 5NEW>3RESO/WON Lets not install Chuck Norris on Jenkins since it confused some people :) [08:24:07] 3Wikimedia / 3Quality Assurance: Activate ChuckNorris Plugin after Jenkins builds - 10https://bugzilla.wikimedia.org/63305#c5 (10Željko Filipin) :'( [09:10:46] (03PS1) 10Tobias Gritschacher: Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 [09:12:54] (03CR) 10Hoo man: [C: 031] Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [09:28:52] (03CR) 10JanZerebecki: [C: 04-1] "The mwext-Wikibase-repo-getentities-api-tests in zuul/layout.yaml needs to be removed, too." [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [09:34:08] (03PS2) 10Tobias Gritschacher: Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 [09:34:28] (03CR) 10Tobias Gritschacher: "@jzerebecki: fixed. thx!" [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [09:38:02] (03CR) 10JanZerebecki: [C: 031] Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [09:54:42] (03CR) 10Hashar: [C: 032] Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [09:58:11] (03Merged) 10jenkins-bot: Run getentities tests together with other API tests [integration/config] - 10https://gerrit.wikimedia.org/r/167540 (owner: 10Tobias Gritschacher) [10:09:38] hashar: https://github.com/bbatsov/rubocop/issues/1389#event-180346242 [10:10:03] zeljkof: sorry have to rush back home. Poke me again this afternoon please :] [10:10:08] and yeah solved :] [10:10:11] * hashar waves [10:24:03] (03CR) 10Zfilipin: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [10:25:06] (03CR) 10Zfilipin: [C: 031] WIP Environment abstraction layer [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [10:27:06] (03CR) 10Zfilipin: [C: 031] "This is already deployed, right? If so, I am fine with merging it." [integration/config] - 10https://gerrit.wikimedia.org/r/166555 (owner: 10Hashar) [11:21:38] 3Wikimedia / 3Quality Assurance: Activate ChuckNorris Plugin after Jenkins builds - 10https://bugzilla.wikimedia.org/63305#c6 (10tobias.gritschacher) Nooooooooooooooooooooooooooooooooooooooooo.....................! :( [11:50:51] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy build #1: SUCCESS in 1 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy/1/ [11:53:13] * zeljkof is out of lunch [12:01:51] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-apertium02.puppetagent.failed_events.value (10.00%) [12:09:26] RECOVERY - BetaLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [12:24:08] * zeljkof is back [12:57:54] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy build #4: FAILURE in 7.3 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy/4/ [13:00:31] Yippee, build fixed! [13:00:32] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy build #5: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-copy/5/ [13:08:14] (03PS1) 10Zfilipin: WIP run "bundle install" before cd-ing into browser folder [integration/config] - 10https://gerrit.wikimedia.org/r/167566 (https://bugzilla.wikimedia.org/69245) [13:43:59] 3Wikimedia / 3Quality Assurance: [Regression] QA: Puppet failing for Role::Ci::Slave::Browsertests/elasticsearch - 10https://bugzilla.wikimedia.org/72255 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None Created attachment 16814 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16814&action=edit Grap... [13:44:24] bd808|BUFFER: hashar: greg-g: ^ [13:44:37] 3Wikimedia / 3Quality Assurance: [Regression] QA: Puppet failing for Role::Ci::Slave::Browsertests/elasticsearch - 10https://bugzilla.wikimedia.org/72255#c1 (10Krinkle) Created attachment 16815 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16815&action=edit Graph: Disk space last week - Nagf The /m... [13:45:23] 3Wikimedia / 3Quality Assurance: [Regression] QA: Puppet failing for Role::Ci::Slave::Browsertests/elasticsearch - 10https://bugzilla.wikimedia.org/72255#c2 (10Krinkle) Created attachment 16816 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16816&action=edit Graph: Puppet runs last week - Nagf Puppe... [13:55:07] hashar: around? [14:08:50] (03PS2) 10Zfilipin: WIP run "bundle install" before cd-ing into browser folder [integration/config] - 10https://gerrit.wikimedia.org/r/167566 (https://bugzilla.wikimedia.org/69245) [14:10:14] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #217: FAILURE in 9.7 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/217/ [14:13:07] Yippee, build fixed! [14:13:07] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #218: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/218/ [14:15:22] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #21: FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/21/ [14:17:47] (03PS3) 10Zfilipin: WIP run "bundle install" before cd-ing into browser folder [integration/config] - 10https://gerrit.wikimedia.org/r/167566 (https://bugzilla.wikimedia.org/69245) [14:18:32] (03PS4) 10Zfilipin: WIP run "bundle install" before cd-ing into browser folder [integration/config] - 10https://gerrit.wikimedia.org/r/167566 (https://bugzilla.wikimedia.org/69245) [14:32:42] (03PS5) 10Zfilipin: WIP run "bundle install" before cd-ing into browser folder [integration/config] - 10https://gerrit.wikimedia.org/r/167566 (https://bugzilla.wikimedia.org/69245) [14:36:49] (03Abandoned) 10Zfilipin: WIP Ignore all rubocop warnings [ruby/api] - 10https://gerrit.wikimedia.org/r/150565 (owner: 10Zfilipin) [14:39:55] (03PS4) 10Zfilipin: Add RuboCop to gemspec file [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (https://bugzilla.wikimedia.org/69245) [14:42:10] (03CR) 10Zfilipin: "check experimental" [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [14:50:09] (03PS1) 10Zfilipin: Set up RuboCop [selenium] - 10https://gerrit.wikimedia.org/r/167578 (https://bugzilla.wikimedia.org/69245) [14:51:17] (03CR) 10Zfilipin: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/167578 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [15:16:55] Yippee, build fixed! [15:16:56] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #22: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/22/ [15:23:06] I'm about to update the salt master for all of labs; salt commands may not work properly for a few moments [15:26:54] done, minions are responding just fine [15:27:10] !log upgrded salt-master on virt1000 (master for labs) [15:27:13] Logged the message, Master [15:27:27] hm I wonder if there's another channel for that, there's a labs channel isn't there [15:28:07] (03PS1) 10Zfilipin: CentralAuth has browser tests [selenium] - 10https://gerrit.wikimedia.org/r/167587 [15:36:43] apergos: changing global labs things should probably be logged in #-operations actually I think. [15:36:58] it will be shortly [15:37:09] along with all the other precise hosts [15:37:22] neat [15:37:41] it's an apergos in -qa! :) [15:37:53] don't get to used to it :-D [15:38:01] you know you wanna [15:38:09] this is enough channels that pidgin autojoin fails [15:38:12] We have successfully tricked another root into testing in beta :) [15:38:15] thank you freenode [15:38:35] as I recall I came to you :-P [15:38:46] You did indeed [15:39:57] (03PS5) 10Zfilipin: Prepare repository for running RuboCop after every push to Gerrit [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (https://bugzilla.wikimedia.org/69245) [15:41:07] (03PS2) 10Zfilipin: Prepare repository for running RuboCop after every push to Gerrit [selenium] - 10https://gerrit.wikimedia.org/r/167578 (https://bugzilla.wikimedia.org/69245) [15:49:34] bd808: what would you recommend abuot lab instances? should I blanket force a salt upgrade on them? [15:49:55] it's not required, maybe some later master upgrade will need it but not this one [15:52:10] apergos: hmmm... maybe a better question for andrewboggott? [15:52:32] ok [15:52:40] I guess he'll be on in a while [15:52:44] Since I don't have access to run salt commands across labs it doesn't matter to me. :) [15:52:59] He's in #-labs [15:56:20] ok [15:56:36] (I'm alredy in there, yet another channel that puts me ovr the autojoin spam limit) [16:36:44] (03CR) 10Legoktm: [C: 032] CentralAuth has browser tests [selenium] - 10https://gerrit.wikimedia.org/r/167587 (owner: 10Zfilipin) [16:36:59] (03Merged) 10jenkins-bot: CentralAuth has browser tests [selenium] - 10https://gerrit.wikimedia.org/r/167587 (owner: 10Zfilipin) [17:28:11] bd808, so, i added additional logging to catch any errors from bunyan libs when writing to stream .. and nothing is being reported, so, not sure where the events get lost in between ... cscott gelf-stream uses udp, right, or is it tcp? [17:28:30] cscott, i can look up if you don't remember offhand. [17:29:32] my network debugging skills are a bit rusty now ...i could use a helping hand from someone (for betalabs). [17:34:18] subbu: You should be able to see if packets are getting to deployment-logstash1 by running something like `sudo tcpdump udp and "dst port 12201"` there [17:34:21] yes, it uses udp [17:34:43] bd808, ok. [17:36:23] And you should be able to run something similar on your log emitting host to see if packets are leaving there headed for the logstash server. [17:36:54] reaching logstash [17:37:06] 17:36:14.093103 IP deployment-parsoid04.eqiad.wmflabs.37077 > deployment-logstash1.eqiad.wmflabs.12201: UDP, length 307 [17:37:06] 17:36:14.202602 IP deployment-parsoid04.eqiad.wmflabs.37077 > deployment-logstash1.eqiad.wmflabs.12201: UDP, length 338 [17:37:06] 17:36:16.755168 IP deployment-parsoid04.eqiad.wmflabs.37077 > deployment-logstash1.eqiad.wmflabs.12201: UDP, length 319 [17:37:33] so, i suppose something in the gelf config then? [17:37:34] subbu: ok. Then I guess the problem is in our logstash config or deploy somehere [17:37:39] *somewhere [17:38:23] gah. the redis noise is horrible now [17:39:06] i am also logging these entries in parsoid.log on betalabs .. you should be able to see them in /data/project/parsoid/parsoid.log if it helps debugging for you. [17:39:22] {"name":"parsoid","hostname":"deployment-parsoid04","pid":26612,"level":30,"type":"info","location":"[info][enwiki/Notability_in_English_Wikipedia?oldid=17569]","msg":"started parsing","time":"2014-10-20T17:36:14.084Z","v":0} [17:39:25] that is an example entry. [17:39:48] !log Disabled puppet on deployment-logstash1 for some live hacking of logstash config [17:39:50] Logged the message, Master [17:41:52] !log Disabled redis input plugin and restarted logstash on deployment-logstash1 [17:41:54] Logged the message, Master [17:43:52] subbu: I got the error log for logstash tamed so maybe we can see a failure message. Can you trigger some log events? [17:44:02] will do. [17:44:26] just did. [17:44:43] 17:44:11.223122 IP deployment-parsoid04.eqiad.wmflabs.37077 > deployment-logstash1.eqiad.wmflabs.12201: UDP, length 306 [17:44:56] that is the last packet from the 3 events that were sent. [17:45:14] hmm... nothing at all in the logstash log :( [17:45:32] yes, i had tailed that earlier .. and didn't see anything either. [17:45:58] so, that is why i was wondering if gelf is swallowing it .. which is what you suspected as well. [17:48:07] afk for a while ... but, here is how you can trigger log events ... open http://en.wikipedia.beta.wmflabs.org/wiki/Special:Random and open it in VE (might need to be logged in and have VE enabled if it is not enabled by default on betalabs) [17:49:16] subbu|lunch: ok. I'll see if I can find anything out [18:03:59] subbu|lunch: i wonder if it's worthwhile dusting off my winston->logstash patch and comparing the packets arriving at logstash? [18:04:16] assuming that the winston->logstash path works (or i could probably dump some sample ocg packets) [18:04:18] subbu|lunch: The messages coming in from your app seem to be setting "type" in the gelf packet payload, or at least logstash is seeing them that way. The type rather than being "gelf" as our logstash config expects is "info" [18:05:27] hm. that type info should really be redundant with the level integer, i suppose [18:05:30] cscott, subbu|lunch: This 'type: "info"' is keeping the log events from matching the filters we have for gelf input that would tag them to be added to logstash. [18:05:36] so it shouldn't hurt to force it to 'gelf' [18:06:01] i assume that the winston->logstash path which ocg is using is setting 'type' to 'gelf' [18:06:43] Or it is unset at all. The input filter would add type:gelf if no type exists in the inbound payload I think [18:07:31] Type does not seem to be an expected top level field in the gelf 1.1 spec -- http://www.graylog2.org/resources/gelf/specification [18:08:28] And "type" definitely has special meaning in the logstash document schema [18:09:20] It would correlate most closely with syslog's "facility" field [18:22:08] bd808|LUNCH, ah, ok .. that should be easy to fix on our end .. cscott although surprised that gelf-stream didn't do that tweaking. [18:23:35] there's a bunyanToGelf function in gelf-stream... wonder if we should submit a github issue? [18:24:34] it seems like bunyanToGelf adds all fields from the bunyan message to gelf except for hostname, time, msg, name, level, and v [18:25:45] according to http://www.graylog2.org/resources/gelf/specification it looks like bunyanToGelf should be prepending those field names with underscore [18:26:04] bd808|LUNCH: i assume that logstash wouldn't have any trouble with a field named _type ? [18:26:54] yes, worth a github issue. [18:31:35] subbu: oh, the gelfling package is doing with underscore prepending -- or shouldbe. [18:31:48] --> #parsoid? [18:32:35] bd808|LUNCH, subbu: can you post/verify a specific JSON message from the UDP stream? the one subbu posted above looks like it comes from the parsoid log, before gelf-stream and gelfling have munged it into gelf format. [18:33:08] I love saying this in bugs: " [18:33:09] Which confirms this is not a "Beta Cluster stability" issue." ;) [18:33:26] bd808|LUNCH: perhaps logstash is undoing the underscore-prepending, and the gelf field is really "_type" already? [18:34:06] i mean, i'm sure we can workaround it just by removing the type field entirely from the parsoid side. but i'd like to get to the bottom of the issue here. [19:01:10] cscott, subbu: Here's a paste taken from the debug logging that I turned on in logstash for a short while -- https://phabricator.wikimedia.org/P27 [19:02:37] bd808: oh, hm -- that doesn't look right. the full_message field shouldn't be the stringified json, the fields should be broken out. shouldn't they? [19:02:59] bd808: if you turn on debugging again, try to get an ocg packet as well, since ocg is "known good" [19:09:51] Yippee, build fixed! [19:09:52] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #242: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/242/ [19:10:30] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #300: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/300/ [19:25:19] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #132: FAILURE in 9 min 48 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/132/ [19:31:08] greg-g: http://shinken.wmflabs.org/all?search=hg:deployment-prep has up/down status for each individual host on deployment-prep :) Service status coming soon [19:31:21] username/password: guest/guest [19:31:33] bd808: ^ :) [19:32:22] for example, http://shinken.wmflabs.org/host/deployment-soa-cache01 is down [19:33:47] no notifications for that yet [19:39:24] YuviPanda: neato, can you send a note to the QA list? [19:39:33] yeah [19:39:45] hmm, I don't know if I'm subscribed. let me go check [19:43:26] greg-g: I've emailed. [19:44:03] greg-g: and you should be personally getting spam now if any other instances go down... [19:47:52] heh, great :) [19:49:19] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #65: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/65/ [19:53:07] bd808, so, on cscott's insistence, i dumped a raw packet dump of what was received on the logstash server end and both of us verified via wireshark that the the nodejs lib we use is sending somethign like: {"host":"deployment-parsoid04","timestamp":1413831675.822,"short_message":"started parsing","facility":"parsoid","level":6,"full_message":"{\n \"name\": \"parsoid\",\n \"hostname\": \"deployment-parsoid04\",\n \"pid\": 26614,\n \"le [19:53:08] vel\": 30,\n \"type\": \"info\",\n \"location\": \"[info][enwiki/0.8848632728114245?oldid=63838]\",\n \"msg\": \"started parsing\",\n \"time\": \"2014-10-20T19:01:15.822Z\",\n \"v\": 0\n}","_pid":26614,"_type":"info","_location":"[info][enwiki/0.8848632728114245?oldid=63838]","version":"1.0"} [19:53:43] so, the field received on logstash end should be "_type" and not "type". [19:54:52] where do beta labs 503 errors get logged? There's no fatals.log in deployment-bastion:/data/project/logs , must I log in to each front-end server? [19:55:02] there are also some differences in how winston-graylog2 is formatting messages compared to bunyan->gelf-stream [19:55:05] subbu: _type and type would be mapped together I think. Those are both magic fields in a way for logstash. [19:55:45] On the app side, what is _type? a severity? a channel? [19:55:53] winston is sending the actual full object as full_message; bunyan->gelfstream is sending a string instead, the JSON string corresponding to the full object. [19:55:55] severity [19:56:09] Reedy: do you know the answer to spagewmf's question? [19:56:22] subbu: severity should be reflected by the 'level' field [19:56:29] yes. [19:56:50] but, bunyan is using the severity info there, it looks like. [19:56:51] * bd808 is debugging a new prod app release and heading into 2 hours of hangouts :( [19:56:52] so we can just remove 'type' from the parsoid message once it gets mapped to level. who does that mapping? [19:57:06] bd808, no rush on this. [19:57:28] fwiw, the field mapping in winston-graylog2 is just: [19:57:30] if (!!meta) { [19:57:30] for (key in meta) { [19:57:30] if (key !== 'id') { [19:57:30] message['_'+key] = meta[key]; [19:57:33] } [19:57:33] } [19:57:35] } [19:58:13] gelf-stream is doing a bunch of flattening as well, it looks like. [19:58:46] that probably doesn't matter, since logstash is flattening at the end. but it might turn out to matter. mental bookmark for later. [20:00:28] subbu: i think my conclusion at this point is that there's not a bug in gelf-stream or gelfling, per se. the only bug might be the underscore-stripping in logstash. but we can work around this by just not sending a field named 'type' in our bunyan adapter, since bunyan is also using `level` as severity. there's no reason to send a type field. [20:00:59] k. [20:41:22] 3Wikimedia / 3Continuous integration: [upstream] Zuul prepareRef does not handle failure to connect to Gearman - 10https://bugzilla.wikimedia.org/72113#c3 (10Antoine "hashar" Musso (WMF)) I have proposed a change at https://review.openstack.org/#/c/128921/ with a test which most probably has a race condition... [20:42:30] 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275 (10spage) 3NEW p:3Unprio s:3normal a:3None I'm getting 503 errors from beta labs when I visit http://en.wikipedia.beta.wmflabs... [20:53:54] 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275 (10Greg Grossmeier) p:5Unprio>3Normal a:3Sam Reed (reedy) [20:54:32] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #347: FAILURE in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/347/ [21:54:26] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #281: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/281/ [22:07:08] 3Wikimedia / 3Continuous integration: Add Jenkins jobs for iegreview application - 10https://bugzilla.wikimedia.org/71617 (10Bryan Davis) 5ASSI>3RESO/FIX [22:38:26] Yippee, build fixed! [22:38:26] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #224: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/224/