[00:01:08] <jgage>	 mforns, are you looking at disk space on eventlog1001? it's almost full.
[00:01:24] <mforns>	 jgage, wow, thanks for the heads up
[00:01:34] <jgage>	 icinga paged me :)
[00:01:55] <mforns>	 jgage, solved
[00:01:59] <jgage>	 thanks amigo
[00:02:02] <mforns>	 sorry for that
[00:02:24] <jgage>	 no problem, glad we caught it
[00:18:44] <nuria>	 madhuvishy: so adding tests might be the way to go to see where things go wrong
[00:20:06] <madhuvishy>	 nuria: yeah. i doubt i'll be able to today, but Dan and I will probably pair on it when he's here thursday. (I'm away until wednesday). But i'm deep into this bug now - hope to figure it out in a bit
[00:20:21] <nuria>	 madhuvishy: sounds good
[00:31:30] <madhuvishy>	 nuria: https://phabricator.wikimedia.org/diffusion/ANWM/browse/master/wikimetrics/models/validate_cohort.py;69b26d41d7b06899e83cda76c75a71f650b37d84$89 here i see that usernames are being inserted. but in the wiki_user table there are both user_names, and associated ids. do you know where/how this association happens?
[00:32:38] <nuria>	 madhuvishy: user_ids are pulled from the wiki project in question "enwiki, dewiki", in vagrant "wiki"
[00:32:50] <madhuvishy>	 nuria: where in the code?
[00:33:28] <nuria>	 madhuvishy: ah , let me look
[00:39:47] <nuria>	 madhuvishy: seems to me that when records are 1st inserted they go in only with "name"
[00:40:20] <madhuvishy>	 nuria: which kinda seems "wrong" when the user supplies ids
[00:40:27] <nuria>	 after the validation process enters the id for the valid ones (and probably non valid too for which it can find ids)
[00:40:50] <nuria>	 madhuvishy: ya, a "little" wrong if that is the on;ly code path
[00:41:11] <madhuvishy>	 nuria: something is going wrong there
[00:41:31] <madhuvishy>	 if the id is not valid
[00:41:58] <madhuvishy>	 the behavior is incorrect
[00:43:55] <nuria>	 madhuvishy: ya, i see several things do not look right
[00:44:32] <nuria>	 the "validate_as_user_ids" field on form
[00:44:53] <nuria>	 is passed along and changes all behaviour: ./wikimetrics/models/validate_cohort.py:
[00:45:01] <madhuvishy>	 nuria: right
[00:45:08] <madhuvishy>	 nuria: and that's intact
[00:45:27] <madhuvishy>	 nuria: it's in the database, and when you validate again, it looks right
[00:45:52] <madhuvishy>	 https://www.irccloud.com/pastebin/Fjywp3wZ
[00:46:00] <madhuvishy>	 this is where it's awry
[00:46:05] <madhuvishy>	 7 and 8 are user ids
[00:46:51] <madhuvishy>	 but validation doesn't put them in mediawiki_userids - which makes sense - they don't really exist
[00:46:59] <madhuvishy>	 but this is wrong too
[00:49:19] <nuria>	 madhuvishy: ya, they are being treated like user_names
[00:50:04] <madhuvishy>	 nuria: this is part of the problem - and then somehow it marks all as invalid user ids - may be its checking the usernames?
[00:50:08] <madhuvishy>	 not sure
[00:50:50] <nuria>	 madhuvishy: i think so, i think user ids are being validated as user names -looks like- and thus they are invalid, sometimes user_names and user_ids are teh same so in instances it might work
[00:51:25] <madhuvishy>	 nuria: Validate again works totally fine if you upload a list of user names.
[00:51:40] <madhuvishy>	 nuria: but yes, for user ids, this is plausible
[00:52:12] <nuria>	 madhuvishy: well, there you go, you might need to refactor the id workflow, but it might be wrong only when it comes from ui, you can write tests and see
[00:52:22] <nuria>	 if that makes sense
[00:52:26] <nuria>	 Ironholds: yt?
[00:52:38] <madhuvishy>	 nuria: yeah.
[00:53:06] <nuria>	 Ironholds: for the apps session job (that BTW it runs on 6 mins now for a whole day)
[00:53:33] <nuria>	 Ironholds: you calculated percentiles ?
[00:53:49] <nuria>	 or rather gave lowbound , highbound for quantiles?
[00:54:00] <nuria>	 this one seems a lot harder to interpret for a pm
[00:55:08] <nuria>	 Ironholds: the phab ticket says "quantiles"
[01:03:08] <nuria>	 Ironholds: doing percentiles, let me know otherwise
[01:03:58] <nuria>	 Ironholds: maybe we just got mixed up with "quaRtiles"  and quaNtiles"
[04:59:17] <jgage>	 i never know what to do when i get these alerts from Icinga: PROBLEM alert - graphite1001/Difference between raw and validated EventLogging overall message rates is CRITICAL
[04:59:41] <jgage>	 CRITICAL: 20.00% of data above the critical threshold [1800.0]
[05:04:46] <YuviPanda>	 jgage: I guess it means that there are way too many invalid events coming in...