[02:09:05] <ori>	 hey Ironholds
[02:09:06] <ori>	 you around?
[02:09:11] <ori>	 or halfak
[02:09:28] <ori>	 or nuria
[02:10:36] <wikibugs>	 Analytics, operations: analytics1013 crashed, investigate... - https://phabricator.wikimedia.org/T97380#1240666 (BBlack) Note that aside from the hung task stuff above, there was no final kernel crash output or anything, and other "normal" logging continues through about 01:47.  When icinga alerted on all...
[02:43:04] <wikibugs>	 Analytics, operations: analytics1013 crashed, investigate... - https://phabricator.wikimedia.org/T97380#1240688 (Ottomata) Yeah this is very strange.  This is the 4th node we have had this happen to in the last 2 weeks or so.  (Well, 1016 happened today, and we are not sure that the same thing happened th...
[03:02:18] <wikibugs>	 Analytics, operations: analytics1013 crashed, investigate... - https://phabricator.wikimedia.org/T97380#1240695 (Ottomata) Note that so far, only the older of the Dells in the cluster have crashed.  analytics1011-analytics1020
[12:37:11] <grrrit-wm>	 (PS1) KartikMistry: Add cs, el, kk and zu languages [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/207063
[13:02:28] <halfak>	 o/ joal & milimetric
[13:02:33] <joal>	 Heya !
[13:31:21] <ottomata>	 joal:  morninnnng!
[13:41:31] <joal>	 ottomata: Moooorning as well !
[13:41:35] <joal>	 wassup ?
[13:41:45] <ottomata>	 impala!  is workkiiing.
[13:42:07] <ottomata>	 it is a bit annoying, have to set REQUEST_POOL (queue name) manually, and have to specificy an impalad to connect to
[13:42:11] <ottomata>	 want to talk to you a bit about that
[13:42:22] <ottomata>	 but, also, how to test it?  like, what use case should we test it out for right now?
[13:42:33] <ottomata>	 impala kinda needs to do the compute stats things on tables before they are useable
[13:42:39] <ottomata>	 i ran that for the wmf.mediacounts table
[13:43:32] <joal>	 is this Give me amin, and let batcave !
[13:43:35] <joal>	 ottomata: --^
[13:44:37] <ottomata>	 ook
[13:45:19] <ottomata>	 there.
[14:50:34] <halfak>	 joal, I just invited you to a meeting with the altiscale folks.  I will be pretty late for you, so don't feel obligated.
[14:50:44] <joal>	 k, thx :)
[14:50:51] <halfak>	 I figured it would be a good chance to push on any open tickets (e.g. spark logging)
[15:33:05] <ottomata>	 joal:  standup!
[15:35:57] <joal>	 Thx ottomata
[15:36:41] <grrrit-wm>	 (CR) KartikMistry: [C: 2] Add cs, el, kk and zu languages [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/207063 (owner: KartikMistry)
[15:36:54] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} [13 pts] - https://phabricator.wikimedia.org/T92977#1241794 (kevinator) a:madhuvishy
[15:44:15] <grrrit-wm>	 (Merged) jenkins-bot: Add cs, el, kk and zu languages [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/207063 (owner: KartikMistry)
[16:00:40] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#1241891 (Tgr) declined>Open Sending a warning message when the logging call fails does not fix the issue o...
[16:09:55] <mforns>	 milimetric, how should I continue with new EL patch?
[16:10:02] <mforns>	 milimetric, shoould I deploy it?
[16:10:39] <mforns>	 milimetric, oh! I'm looking at your comments
[16:40:31] <mforns>	 milimetric, added some comments, thanks!
[16:52:40] <ottomata>	 changing locations, back in a bit
[17:08:19] <milimetric>	 mforns: checking now, how long are you around?
[17:08:35] <mforns>	 milimetric, for 5 more hours
[17:14:30] <wikibugs>	 Analytics-Engineering, MediaWiki-API, Wikipedia-Android-App, Wikipedia-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1242164 (bd808)
[17:31:59] <milimetric>	 mforns: ok, merged, good work.  We should deploy it?
[17:32:02] <milimetric>	 I've never deployed...
[17:32:18] <mforns>	 milimetric, did you apply any changes?
[17:35:12] <milimetric>	 mforns: no, agreed with your comments
[17:35:28] <mforns>	 milimetric, I also never deployed
[17:35:46] <milimetric>	 ok :) so let's read the docs and meet up in the batcave afterwards and try it together?
[17:36:12] <mforns>	 I suppose that puppet pulls latest changes to the repo, so the only thing we'd need is execute setup script and restart?
[17:38:30] <mforns>	 milimetric, the repo is not up to date yet
[17:38:51] <mforns>	 milimetric, cool! didn't get your last message
[17:43:43] <milimetric>	 mforns: i can't find any documentation, you know where it is?
[17:43:55] <mforns>	 milimetric, https://wikitech.wikimedia.org/wiki/EventLogging#Deploying_EventLogging
[17:46:58] <milimetric>	 :) I was looking on mediawiki of course
[17:47:03] <milimetric>	 that doc sprint can't come soon enough
[17:48:22] <joal>	 ottomata: Do we have access to spark 1.3 in our codebase ?
[17:49:12] <milimetric>	 ottomata: mforns and I are about to try to deploy EL.  Anything we should know since it's been moved to eventlog1001?  Does the process still go roughly as outlined in https://wikitech.wikimedia.org/wiki/EventLogging#Deploying_EventLogging ?
[17:49:23] <ottomata>	 milimetric: process should be the same
[17:49:32] <ottomata>	 joal: it is not officially instaleld, but I do have it in my homedir on stat1002
[17:49:36] <ottomata>	 you can use it
[17:49:37] <milimetric>	 I can update small differences, but we do git deploy from tin, then go to eventlog1001 and hafnium and build, start, stop
[17:49:48] <mforns>	 milimetric, aha
[17:49:51] <ottomata>	 yes
[17:49:59] <milimetric>	 ok, cool, mforns you wanna jump in the batcave now and do it?
[17:50:01] <mforns>	 milimetric, batcave?
[17:50:03] <joal>	 ottomata: https://issues.apache.org/jira/browse/SPARK-4987
[17:50:03] <mforns>	 ok
[17:50:05] <milimetric>	 k :)
[17:50:12] * joal facepalm
[17:50:35] <ottomata>	 oh yeah
[17:50:48] <ottomata>	 what's that do to us with current spark?  we can't read the table at all?
[17:50:56] <joal>	 nope
[17:51:06] <ottomata>	 doh, ok.
[17:51:06] <joal>	 breaks completely on my side
[17:51:07] <ottomata>	 heh
[17:51:17] <joal>	 I wonder how Nuria has managed it ...
[17:51:20] <ottomata>	 ok, welp, i guess I should install 1.3 more officially then, outside of packages.
[17:51:30] <ottomata>	 you are using sqlContext.parquetFile?
[17:51:33] <joal>	 Would help for sure
[17:51:41] <ottomata>	 maybe if you only select certain fields?
[17:51:46] <joal>	 Sorry man :-S
[17:51:57] <joal>	 Well, I don't even look at timestamp
[17:52:04] <ottomata>	 hm
[17:53:53] <joal>	 how should I use your spark for spark-shell ?
[17:53:55] <joal>	 hmmm. probably only for spark-submit I guess
[17:54:30] <ottomata>	 you can do it with spark-shell
[17:54:30] <ottomata>	 um
[17:54:45] <ottomata>	 cd /home/otto/spark-1.3.0-bin-hadoop2.4
[17:54:47] <ottomata>	 bin/spark-shell
[17:54:49] <ottomata>	 i think that should do it
[17:54:53] <joal>	 cool
[17:55:26] <ottomata>	 uhhh
[17:55:28] <ottomata>	 but i just did that
[17:55:30] <ottomata>	 and got java.lang.UnsupportedOperationException: Parquet does not support timestamp. See HIVE-6384
[17:56:25] <joal>	 hive 0.14 :(
[17:56:48] <ottomata>	 oof right
[17:57:04] <ottomata>	 hm, well, hm
[17:57:09] <ottomata>	 wonder if that is just spark then
[17:57:13] <ottomata>	 maybe we can put 0.14 on classpath
[17:57:15] <ottomata>	 trying...
[17:57:46] <joal>	 arrrf, that's bad, even if you don't read the thing, it breaks :(
[17:59:23] <ottomata>	 hm, wait no, spark-assembly.jar has hive . parquet classes in it
[17:59:43] <ottomata>	 would assume that would mean 1.3 would have hive 0.14 deps included
[18:01:51] <ottomata>	 haha, joal, or maybe we should just remove ts for now????? :p
[18:01:59] <ottomata>	 oof
[18:02:04] <joal>	 Yeah ... I wonder as well :)
[18:02:05] <ottomata>	 sorry bout that
[18:02:25] <joal>	 It's painfull, but is it as much as not having data workable ?
[18:02:49] <joal>	 Biggest concern here : would have to replicate almost all of april data :(
[18:02:56] <joal>	 mrff
[18:03:02] <ottomata>	 oh, ha joal
[18:03:03] <ottomata>	 http://blog.cloudera.com/blog/2015/04/cloudera-enterprise-5-4-is-released/
[18:03:07] <ottomata>	 reading to see if it has what we need
[18:03:15] <ottomata>	 	•	Apache Spark 1.3
[18:03:17] <ottomata>	 	•	Apache Hive 1.1
[18:03:20] <ottomata>	 should be good!
[18:03:24] <joal>	 Ok, Indeed !
[18:03:58] <joal>	 Man ... That timestamp stuff was really a bad mistake ...
[18:04:07] <ottomata>	 i mean, not quite a mistake
[18:04:10] <ottomata>	 it was the right decisison!
[18:04:19] <ottomata>	 we are just ahead of the times :)
[18:04:22] <joal>	 I hope it'll work smoothly to update to 5.4 !
[18:04:37] <joal>	 huhuhu --> No regression test, that's not so good ;)
[18:05:15] <joal>	 Let me know if you need help on that
[18:05:42] <ottomata>	 k, reading release notes now
[18:07:06] <milimetric>	 ottomata: so we started git deploy sync from tin, but it said 0/4 minions completed the fetch
[18:07:10] <ottomata>	 also just in time!
[18:07:10] <ottomata>	 Added Spark action which lets you run Spark applications from Oozie workflows
[18:07:15] <ottomata>	 joal^
[18:07:18] <ottomata>	 hm, milimetric, weird
[18:07:21] <milimetric>	 then we checked EL on eventlog1001 and hafnium and it's in detached head
[18:07:27] <ottomata>	 hmm
[18:07:32] <joal>	 ottomata: Hurroay :)
[18:07:45] <milimetric>	 even weirder, what's checked out on hafnium is different from what's on eventlog1001
[18:07:52] <joal>	 ottomata: as you said, jsut ahead of time, one version !
[18:08:20] <milimetric>	 ottomata: should we just check out master on both machines and try git deploy sync again?
[18:08:30] <ottomata>	 detached head is fine, right?  that's how git deploy works?
[18:08:31] <ottomata>	 not sure.
[18:08:34] <milimetric>	 (we're in the batcave trying to figure this out if that's easier)
[18:08:36] <ottomata>	 um, yeah, milimetric that won't hurt
[18:08:38] <ottomata>	 that's what I would try
[18:08:41] <milimetric>	 k
[18:09:46] <milimetric>	 ottomata: it says hafnium is 351 commits behind!
[18:09:51] <milimetric>	 what role does that box play?
[18:10:09] <ottomata>	 awww, joal
[18:10:09] <ottomata>	 The following is not yet supported in a production environment because of its immaturity:
[18:10:09] <ottomata>	 	•	Spark SQL (which now includes dataframes)
[18:10:17] <ottomata>	 milimetric: i think it is for monitoring?
[18:10:25] <joal>	 :(((
[18:10:26] <milimetric>	 k, thx
[18:10:31] <joal>	 ottomata: --^
[18:11:58] <ottomata>	 maybe it is included, but just not officially cloudera supported, joal
[18:12:05] <ottomata>	 they said the same for 1.2
[18:12:08] <ottomata>	 and we use it :)
[18:13:54] <joal>	 ottomata: I think that's what it is, but still
[18:14:12] <joal>	 ayway, let's try, it should solve the hive timestamp issue as well
[18:15:32] <ottomata>	 k, lemme finish up something, will go through upgrade in vagrant and then in labs and see how it goes
[18:15:38] <ottomata>	 milimetric: how goes?
[18:15:48] <joal>	 ottomata: You are my savior !
[18:15:49] <joal>	 ;)
[18:16:03] <milimetric>	 ottomata: we just merged on hafnium, git deploy had done the fetch part
[18:16:09] <milimetric>	 and i guess just mis-reported that it failed
[18:16:20] <milimetric>	 the merge went ok now we're checking graphite to see that it didn't break anything
[18:17:10] <ottomata>	 milimetric: sometimes with git deploy i wait and ask it to retry, wait a few secs and show report again
[18:17:16] <ottomata>	 sometimes it just takes longer than git deploy wants
[18:17:21] <milimetric>	 gotcha
[18:33:17] <milimetric>	 ottomata: hafnium says "fetch status: 128" in git deploy
[18:33:50] <milimetric>	 and I get "fatal: loose object" when I try to pull from hafnium
[18:34:37] <ottomata>	 sheesh y u gotta have problems?!
[18:34:41] <ottomata>	 with you in 1 min...
[18:35:53] <ottomata>	 milimetric:  can I poke around?
[18:36:05] <milimetric>	 ottomata:  sure, come to the batcave
[18:42:37] <ottomata>	 joal, ja, known fix in 5.4: — Hive's Decimal type cannot be stored in Parquet and Avro
[18:43:03] <joal>	 ottomata: ok, but timestamps ?
[18:43:13] <ottomata>	 oh
[18:43:14] <ottomata>	 sorry
[18:43:24] <joal>	 ;)
[18:44:05] <ottomata>	 ah, from 5.3.2
[18:44:05] <ottomata>	 yes
[18:44:14] <joal>	 nice :)
[18:48:20] <wikibugs>	 Analytics-Cluster: Upgrade Analytics Cluster to CDH 5.4.0 - https://phabricator.wikimedia.org/T97453#1242768 (Ottomata) NEW a:Ottomata
[18:51:57] <joal>	 Time for me to say goodnight !
[18:52:05] <joal>	 Will catch up with you ottomata tomorrow
[18:52:57] <ottomata>	 goodnight!
[18:53:52] <mforns>	 milimetric, did you !log the eventlogging1001 deployment?
[18:56:40] <milimetric>	 mforns: yes, in -ops
[18:56:58] <mforns>	 milimetric, ok!
[20:24:12] <halfak>	 Hey folks.  It looks like EL is about 2 hours behind.
[20:24:19] <halfak>	 FYI.
[20:24:25] * halfak waits for his events to show up. 
[20:52:21] <mforns>	 milimetric, eventlogging is crashing
[20:54:15] <milimetric>	 mforns: uh oh
[20:54:41] <mforns>	 milimetric, sudo tail -f /srv/log/upstart/eventlogging_consumer-mysql-m4-master.log
[20:57:45] <milimetric>	 mforns: hm... so somehow invalid events are getting past the processor?
[20:57:51] <milimetric>	 or something's wrong with the consumer you think?
[20:58:03] <mforns>	 milimetric, I don't know...
[20:58:11] <milimetric>	 all-events.log seems to be showing events
[20:58:20] <mforns>	 milimetric, aha
[20:58:24] <milimetric>	 graphite looks like it has a downward trajectory but didn't seem to crash yet
[20:59:07] <milimetric>	 it looks normal historically actually
[21:01:05] <mforns>	 milimetric, yes
[21:01:22] <mforns>	 milimetric, I think the problem is in schema.py http_get_schema
[21:02:02] <mforns>	 milimetric, line 65
[21:02:20] * milimetric looking
[21:03:28] <milimetric>	 i forget if you changed that scid in your patch
[21:03:38] <mforns>	 milimetric, like the retrieved schema was not valid
[21:03:47] <milimetric>	 yeah, it looks like it
[21:04:58] <mforns>	 milimetric, I'm going to change the code in production to print the schema when the schema validation raises error, makes sense?
[21:05:16] <milimetric>	 yeah
[21:05:19] <mforns>	 ok
[21:05:25] <milimetric>	 so I'm looking at this line:
[21:05:25] <milimetric>	 scid, scid_events = events_batch.pop()
[21:05:39] <milimetric>	 would that scid be incompatible with what get_schema(scid) is looking for?
[21:05:50] <milimetric>	 sorry
[21:05:57] <milimetric>	 get_table(meta, scid)
[21:08:41] <mforns>	 milimetric, do not follow
[21:08:52] <milimetric>	 mforns: the all-events log stopped
[21:09:12] <mforns>	 milimetric, I restarted eventloggingctl
[21:09:32] <mforns>	 but should be running now
[21:09:50] <milimetric>	 hm, I don't see events on it
[21:10:29] <milimetric>	 yeah, graphite's showing the dip too
[21:10:47] <milimetric>	 we still have the raw logs so that's ok
[21:12:44] <milimetric>	 mforns: yeah, let's go in the batcave
[21:12:49] <mforns>	 ok
[21:14:31] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 40.00% of data above the critical threshold [1800.0]
[21:51:59] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL Stopped EventLogging jobs: reporter/statsd consumer/server-side-events-log consumer/mysql-m4-master consumer/client-side-events-log consumer/client-side-events-kafka-log consumer/all-events-log multiplexer/all-events processor/server-side-events processor/client-side-events-kafka processor/client-side-events forwarder/8422 forwarder/8421
[21:59:20] <mforns>	 milimetric, this is the url that EL uses to get schemas: http://meta.wikimedia.org/w/api.php?action=jsonschema&title=MobileWikiAppArticleSuggestions&revid=11448426
[21:59:32] <mforns>	 milimetric, required comes already empty
[22:02:57] <mforns>	 milimetric, maybe this change: https://gerrit.wikimedia.org/r/#/c/207274/1
[22:18:35] <mforns>	 kevinator, yt? do you know who from mediawiki-core I can contact for a problem in mediawiki API that is breaking EventLogging?
[22:19:09] * YuviPanda is curious too, re ^ - now that mediawiki-core does not exist
[22:19:38] <mforns>	 YuviPanda, hehe do you know someone that can help me?
[22:19:53] <YuviPanda>	 mforns: usually, if in doubt ask bd808 and he’ll point you to someone
[22:19:58] <YuviPanda>	 that applies to life in general
[22:20:05] <mforns>	 YuviPanda, xD ok
[22:20:08] <mforns>	 thanks!
[22:20:46] <bd808>	 mforns: what
[22:20:51] <bd808>	 what's up?
[22:20:52] <mforns>	 bd808, I think there's a recently deployed problem with mediawiki API
[22:20:58] <mforns>	 hi bd808!
[22:21:11] <bd808>	 probably. Brad's been cleaning up a couple of things today
[22:21:31] <bd808>	 wmf3 had some bigish changes apparently
[22:21:37] <mforns>	 this url was returning boolean true values normally as of 2 hours ago
[22:21:55] <mforns>	 bd808, http://meta.wikimedia.org/w/api.php?action=jsonschema&title=MobileWikiAppArticleSuggestions&revid=11448426
[22:22:13] <mforns>	 look at the 'required' fields
[22:23:11] <mforns>	 bd808, they are empty, and if you look at the corresponding page: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppArticleSuggestions
[22:23:17] <mforns>	 they should be true
[22:24:09] <mforns>	 bd808, thanks for pointing me to him, I already contacted him, because I thought the problem had something to do with: https://gerrit.wikimedia.org/r/#/c/207274/1, but he seems out
[22:25:10] * bd808 had a wifi drop
[22:25:47] <mforns>	 bd808, can you read the previous messages?
[22:25:51] <bd808>	 Brad is off for the day by this time generally. He's on EDT (3 hours ahead of SF)
[22:26:07] <bd808>	 yeah. znc for the win :)
[22:26:24] <mforns>	 bd808, ok, the problem is completely breaking one of our main systems..
[22:27:05] <bd808>	 there were changes to the API output formatting. let me see if I can find somebody to help track this down
[22:27:18] <mforns>	 bd808, thanks!
[22:32:21] <bd808>	 mforns: legoktm is going to help track this down. He's got a good idea what's wrong
[22:32:30] <bd808>	 have you opened a bug yet?
[22:32:43] <mforns>	 bd808, thanks for the help
[22:32:51] <mforns>	 bd808, no, no task yet
[22:32:55] <YuviPanda>	 see, ask bd808 and all problems go away :)
[22:33:07] <bd808>	 mforns: can you write one up pretty please?
[22:33:11] <mforns>	 bd808, do you know in which project should I create that
[22:33:18] <mforns>	 bd808, sure!
[22:33:23] <bd808>	 what owns the jsonschema module?
[22:34:18] <legoktm>	 mforns: hi, are there any other keys besides 'required' that are being affected?
[22:34:31] <YuviPanda>	 (JsonSchema* is in EventLogging)
[22:34:38] <mforns>	 legoktm, hi! I don't know
[22:34:52] <bd808>	 mforns: tag the bug with EventLogging and mediawiki-api
[22:34:54] <mforns>	 legoktm, it seems that it only happens when the original value is true
[22:35:02] <mforns>	 bd808, sure
[22:35:03] <legoktm>	 yes, it does that for any boolean
[22:35:09] <mforns>	 ok
[22:36:04] <legoktm>	 we need a whitelist of all the acceptable boolean parameters
[22:36:12] <legoktm>	 https://github.com/wikimedia/mediawiki-extensions-TemplateData/commit/6580dc3e872d20750c8ef624299af35952caefa9 is how we fixed this in a different extension
[22:37:37] <bd808>	 delightful that the fix is a "hack" :/
[22:40:04] <mforns>	 legoktm, this would mean that everytime a team creates a schema with a new boolean value, we need to ping you to add a new field to that whitelist?
[22:40:13] <legoktm>	 umm
[22:40:27] <legoktm>	 :|
[22:40:36] <legoktm>	 where's the code that's accessing this?
[22:40:54] <legoktm>	 it might just be easier to switch to formatversion=2...
[22:41:19] <legoktm>	 https://meta.wikimedia.org/w/api.php?action=jsonschema&title=MobileWikiAppArticleSuggestions&revid=11448426&formatversion=2
[22:41:23] <mforns>	 legoktm, https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/schema.py#L62
[22:41:39] <legoktm>	 okay nope :P
[22:42:22] <mforns>	 legoktm, formatversion=2 seems to work!
[22:43:32] <mforns>	 legoktm, is formatversion=2 a deprecated version?
[22:43:42] <legoktm>	 it's alpha
[22:43:44] <bd808>	 legoktm: can jsut change this right? -- https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/schema.py#L31
[22:44:13] <milimetric>	 yeah, we can change to formatversion=2 but if it's alpha we'd probably rather think some more, since this is fairly critical
[22:44:27] <mforns>	 legoktm, bd808, yes we could easily change that to use formatversion=2
[22:44:39] <mforns>	 hi milimetric
[22:44:42] <milimetric>	 hi :)
[22:44:45] <milimetric>	 was just catching up
[22:45:00] <milimetric>	 is it wrong that I feel relieved we didn't screw this up this time? :)
[22:45:06] <legoktm>	 I don't think that's a good idea yet
[22:45:07] <legoktm>	 https://gerrit.wikimedia.org/r/207297
[22:45:29] <legoktm>	 how do we figure out if any other keys were using raw booleans?
[22:45:42] <YuviPanda>	 mwsearch?
[22:45:50] <YuviPanda>	 there aren’t many Schema: things
[22:45:59] <legoktm>	 you mean mwgrep? wouldn't really help here....
[22:46:00] <milimetric>	 legoktm: a bunch of other keys do it, basically EventLogging allows any schema to have any key that uses booleans
[22:46:06] <legoktm>	 hmm
[22:46:10] <milimetric>	 so, essentially, the list of fields is dynamic
[22:46:10] <legoktm>	 milimetric: do you have an example?
[22:46:21] <milimetric>	 sure, sec
[22:47:03] <milimetric>	 legoktm: https://meta.wikimedia.org/wiki/Schema:NavigationTiming look for isAnon
[22:47:03] <mforns>	 legoktm, https://meta.wikimedia.org/wiki/Schema:MultimediaViewerNetworkPerformance
[22:47:22] <milimetric>	 but that's not the point, EventLogging allows anyone to create a new schema with fields of type boolean
[22:47:35] <legoktm>	 umm
[22:47:42] <legoktm>	 both of those only have 'required' as boolean
[22:48:03] <bd808>	 the schema would mark it as type boolean right?
[22:48:05] <milimetric>	 doh
[22:48:12] <legoktm>	 I'm asking in the JSON of the schema, which of those fields are a boolean. Not in the data the schema represents
[22:48:18] <milimetric>	 right, the events themselves would have raw booleans
[22:48:19] <milimetric>	 sorry
[22:48:29] <milimetric>	 uh.... yeah, i guess right now it's just required
[22:48:55] <legoktm>	 ok, then https://gerrit.wikimedia.org/r/207297 will fix it :)
[22:49:03] <legoktm>	 is there a bug for this?
[22:49:13] <legoktm>	 and is this unbreak now or?
[22:49:14] <milimetric>	 legoktm: what's the timing on that, when could we expect to see good data again?
[22:49:24] <milimetric>	 well, EventLogging is down without some kind of fix
[22:49:28] <bd808>	 it's UBN
[22:49:33] <milimetric>	 so we need to know if we should do formatversion=2
[22:50:14] <legoktm>	 as soon as I can find someone to +2 it I can deploy it
[22:50:19] <bd808>	 can you guys recreate the problem in beta? We can merge, see the fix there and then backport to wmf3
[22:50:47] <milimetric>	 bd808: checking beta now
[22:50:53] <milimetric>	 mforns: feel free to go to sleep, I'm ok with this
[22:51:05] <mforns>	 milimetric, ok, just creating a task
[22:51:10] <milimetric>	 cool
[22:51:33] <bd808>	 legoktm: +2 {{done}}
[22:51:43] <legoktm>	 thanks
[22:51:51] * bd808 should have waited for mforns to reproduce
[22:52:23] <legoktm>	 I reproduced and tested it locally
[22:53:24] <mforns>	 legoktm, bd808, so no phab task needed?
[22:53:28] <milimetric>	 legoktm / bd808: if we deploy this to beta, and events start showing up in the validated log, then the issue is fixed
[22:53:40] <bd808>	 mforns: we should have a task
[22:53:41] <milimetric>	 right now, no events are getting added to that log
[22:53:41] <legoktm>	 uh, please file a bug for this
[22:53:47] <mforns>	 sure
[22:54:48] <mforns>	 legoktm, bd808, milimetric, here: https://phabricator.wikimedia.org/T97487
[22:56:24] <mforns>	 added the wrong reponses to the task description
[22:56:48] <milimetric>	 legoktm / bd808: hm, beta seems fine now, events were coming in when I hit some pages
[22:57:02] <milimetric>	 but i'm unclear if this breaking API change was deployed there
[22:57:15] <bd808>	 milimetric: we merged the core change. it should be deployed there now
[22:57:22] <milimetric>	 ah, ok
[22:57:29] <bd808>	 I'll verify though
[22:58:36] <bd808>	 milimetric: confirmed that "6c0ae4d API: Force 'required' key to use bools in formatversion=1" is the HEAD of extensions/EventLogging in beta cluster now
[22:58:59] <legoktm>	 https://meta.wikimedia.org/w/api.php?action=jsonschema&title=MobileWikiAppArticleSuggestions&revid=11448426 looks good again
[22:59:15] <bd808>	 also... you guys should have some tests in beta for this sort of thing. It would have been broken there for >1 week
[22:59:17] <mforns>	 \o/
[22:59:45] <milimetric>	 bd808: odd, we did test in beta a bunch
[22:59:58] <bd808>	 thanks tons legoktm :)
[23:00:04] <legoktm>	 :)
[23:00:09] <milimetric>	 thanks bd808 & legoktm
[23:00:17] <mforns>	 thanks guys!
[23:00:38] <milimetric>	 mforns: so i'll undo your change to schema and restart
[23:00:54] <mforns>	 milimetric, I think the mediawiki code was deployed after our deploy today
[23:01:00] <mforns>	 ok
[23:01:10] <milimetric>	 yeah, but it was supposed to be in beta for a while, is what I think bd808 said
[23:01:18] <milimetric>	 and you tested there over the last two days right?
[23:01:21] <mforns>	 milimetric, I see
[23:01:33] <legoktm>	 does beta eventlogging use meta.wikimedia.org for the API? or beta cluster's metawiki?
[23:01:43] <mforns>	 milimetric, yes, but pointing to the normal API
[23:01:54] <milimetric>	 oh
[23:02:06] <milimetric>	 that's the mismatch then, yea
[23:02:44] <milimetric>	 yeah, we should see if we can make beta tests more thorough
[23:02:48] <milimetric>	 and maybe continuous :)
[23:03:08] <milimetric>	 mforns: events are coming into all-events.log again, all is well
[23:03:09] <legoktm>	 even then it should have been caught on test2wiki
[23:03:10] <milimetric>	 thanks!
[23:03:18] <mforns>	 milimetric, good!
[23:03:20] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK All defined EventLogging jobs are runnning.
[23:03:21] <milimetric>	 legoktm: what's test2wiki?
[23:03:31] <legoktm>	 test2.wikipedia.org ?
[23:03:40] <legoktm>	 	if ( $wgDBname === 'test2wiki' ) {
[23:03:40] <legoktm>	 		// test2wiki has its own Schema: NS.
[23:03:40] <legoktm>	 		$wgEventLoggingDBname = 'test2wiki';
[23:03:40] <legoktm>	 		$wgEventLoggingSchemaApiUri = 'http://test2.wikipedia.org/w/api.php';
[23:03:41] <legoktm>	 		$wgEventLoggingBaseUri = "//{$wmfHostnames['bits']}/dummy.gif";
[23:03:43] <legoktm>	 		$wgEventLoggingFile = "udp://$wmfUdp2logDest/EventLogging-$wgDBname";
[23:03:55] <milimetric>	 hm, i don't think we test there, i've never heard of it
[23:04:15] <legoktm>	 well it's explicitly configured for you to test there...
[23:04:18] <milimetric>	 if you have a sec to explain how that works or if you have any docs, that'd be great
[23:04:49] <legoktm>	 code will be deployed on test2wiki on wednesdays and then it will hit metawiki on tuesday
[23:05:05] <milimetric>	 hm, odd, why would we ever test on beta then...
[23:05:16] <legoktm>	 beta is master, and test2wiki is still release branches
[23:05:33] <milimetric>	 seems like test2 would let us catch more problems earlier then
[23:05:58] <milimetric>	 like, it's a little late if the problem hits master...
[23:06:34] <legoktm>	 this should have never hit production, it should have been caught in beta
[23:07:00] <milimetric>	 you mean for the API change, right?
[23:07:03] <legoktm>	 yes
[23:07:08] <milimetric>	 on our side we should configure beta to use the beta API though
[23:07:17] <milimetric>	 that would've helped and then maybe we could've helped you find the bug
[23:07:22] <milimetric>	 because we would've seen it last Friday
[23:07:31] <milimetric>	 or just test in test2
[23:09:15] <mforns>	 good night everyone, see you tomorrow :]
[23:09:26] <milimetric>	 nite
[23:11:09] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [1200.0]
[23:45:03] <kevinator>	 halfak: milimetric's computer just crashed :-(
[23:45:18] <kevinator>	 he is rebooting
[23:45:24] <halfak>	 kevinator, thanks.  Good to know I didn't miss him.