[02:01:48] <wikibugs>	 Analytics, MediaWiki-Authentication-and-authorization, Reading-Infrastructure-Team, MW-1.26-release, and 2 others: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1513464 (Tgr)
[04:10:17] <grrrit-wm>	 (PS1) Ori.livneh: Lower the log level of stats output to stderr [analytics/statsv] - https://gerrit.wikimedia.org/r/229623
[04:10:47] <grrrit-wm>	 (CR) Ori.livneh: [C: 2 V: 2] Lower the log level of stats output to stderr [analytics/statsv] - https://gerrit.wikimedia.org/r/229623 (owner: Ori.livneh)
[07:40:23] <wikibugs>	 Analytics-Tech-community-metrics, ECT-August-2015: Tech community KPIs for the WMF metrics meeting - https://phabricator.wikimedia.org/T107562#1513750 (Qgil) The casual presentation yesterday at the weekly WMF Engineering management meeting was interesting. They didn't dig much into the numbers because th...
[07:47:18] <wikibugs>	 Analytics-Tech-community-metrics, Engineering-Community: Automated generation of repositories for Korma - https://phabricator.wikimedia.org/T104845#1513769 (Qgil)
[07:47:20] <wikibugs>	 Analytics-Tech-community-metrics, ECT-August-2015: Jenkins-mwext-sync appears in "Who contributes code" - https://phabricator.wikimedia.org/T105983#1513768 (Qgil)
[07:47:23] <wikibugs>	 Analytics-Tech-community-metrics, Engineering-Community: Check whether it is true that we have lost 40% of code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1513771 (Qgil)
[08:00:52] <wikibugs>	 Analytics-Tech-community-metrics: Present most basic community metrics from T94578 on one page - https://phabricator.wikimedia.org/T100978#1513777 (Qgil) I'm linking this task to T107562 because both a related: we need a good place to highlight and track our KPIs. Options: a special page in Korma or https://...
[08:02:19] <wikibugs>	 Analytics-Tech-community-metrics, ECT-August-2015: Jenkins-mwext-sync appears in "Who contributes code" - https://phabricator.wikimedia.org/T105983#1513780 (Qgil) This is probably affecting one of our KPIs to be presented at the next Metrics meeting, see T107562.
[08:11:33] <wikibugs>	 Analytics-Tech-community-metrics, ECT-August-2015: "Median time to review for Gerrit Changesets, per month": External vs. WMF/WMDE/etc patch authors - https://phabricator.wikimedia.org/T100189#1513789 (Qgil) At least from the point of view of the WMF Engineering management, having reliable metrics by affi...
[08:12:14] <wikibugs>	 Analytics-Tech-community-metrics: Remove deprecated repositories from korma.wmflabs.org code review metrics - https://phabricator.wikimedia.org/T101777#1513792 (Qgil)
[08:13:33] <wikibugs>	 Analytics-Tech-community-metrics: Remove deprecated repositories from korma.wmflabs.org code review metrics - https://phabricator.wikimedia.org/T101777#1347475 (Qgil) Not only clearly deprecated repositories (as in they cannot be found in Gerrit anymore). I think unmaintained repositories should be also remo...
[08:21:42] <wikibugs>	 Analytics-Tech-community-metrics, ECT-August-2015: Tech community KPIs for the WMF metrics meeting - https://phabricator.wikimedia.org/T107562#1513808 (Qgil) I also wonder whether we should highlight the metrics specific to the MediaWiki Core repository. For several reasons:  * It is easier to compare a s...
[11:36:33] <wikibugs>	 Analytics-Kanban: Create Hadoop Job to load data into cassandra [?pts] {slug} - https://phabricator.wikimedia.org/T108174#1514229 (JAllemandou) NEW a:JAllemandou
[12:30:27] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[12:32:36] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]
[14:02:16] <milimetric>	 joal: I'm confused about this routing
[14:02:25] <joal>	 Hi milimetric :)
[14:02:29] <joal>	 cave ?
[14:02:32] <milimetric>	 It looks like I'm doing the same thing as page_revisions but it's not working
[14:02:32] <milimetric>	 k
[14:26:03] <wikibugs>	 Analytics-Tech-community-metrics: Remove deprecated repositories from korma.wmflabs.org code review metrics - https://phabricator.wikimedia.org/T101777#1514563 (Aklapper) >>! In T101777#1513792, @Qgil wrote: > Not only clearly deprecated repositories (as in they cannot be found in Gerrit anymore).   "depreca...
[14:30:23] <wikibugs>	 Analytics-Kanban, Reading-Admin, Research-and-Data, Research consulting: Request for data: sites traffic by topics/ subject areas and geographies - https://phabricator.wikimedia.org/T107613#1514569 (DarTar) a:ezachte
[14:30:42] <wikibugs>	 Analytics-Kanban, Reading-Admin, Research-and-Data, Research consulting: Request for data: sites traffic by topics/ subject areas and geographies - https://phabricator.wikimedia.org/T107613#1499257 (DarTar) p:Triage>Low
[15:12:48] <wikibugs>	 Analytics-Tech-community-metrics, Epic, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1514652 (NiharikaKohli) Hello!  End of GSoC is fast approaching. 17 August is "Suggested pencils down" deadline and 21 A...
[15:20:07] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[15:20:10] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Check and potentially timebox limn-extdist-data reports [3 pts] {tick} - https://phabricator.wikimedia.org/T107506#1514672 (kevinator) Open>Resolved
[15:22:15] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]
[15:33:23] <joal>	 milimetric: managed to log :)
[15:33:29] <joal>	 will let you nkow how
[15:33:31] <milimetric>	 YES!
[15:33:31] <milimetric>	 :)
[15:42:58] <wikibugs>	 Analytics-Kanban, RESTBase-API: create second RESTBase endpoint [8 pts] {slug} - https://phabricator.wikimedia.org/T107054#1485473 (Milimetric)
[15:43:20] <wikibugs>	 Analytics-Kanban, RESTBase-API: create third RESTBase endpoint [8 pts] {slug} - https://phabricator.wikimedia.org/T107055#1485480 (Milimetric)
[15:46:14] <joal>	 ottomata: I don't underdstand why I can't run the aggregator manually :(
[15:46:21] <ottomata>	 oh?
[15:46:22] <ottomata>	 wassup?
[15:46:23] <joal>	 Well, in fact
[15:46:32] <joal>	 I can run the thing (as stats)
[15:46:39] <joal>	 But can't push using git
[15:46:45] <joal>	 any idea on that?
[15:47:25] <joal>	 I cheated a bit: I sudoed a bash
[15:47:40] <joal>	 then ran my things --> no chance
[15:47:45] <joal>	 ottomata: --^
[15:51:19] <ottomata>	 yeah, i thikn there are issues with sudo and git defaults being read from homedirs
[15:51:28] <joal>	 hm
[15:51:29] <ottomata>	 i can do it if I sudo to root and then su stats
[15:51:39] <ottomata>	 but sudo -u stats doesn't seem to work i thikn
[15:51:41] <joal>	 ok weird
[15:52:42] <ottomata>	 hm, maybe we need something like
[15:52:45] <joal>	 Can you run a thing for me ?
[15:52:55] <ottomata>	  /a/aggregator/scripts/bin/git-pusher
[15:52:57] <ottomata>	 that does
[15:53:07] <ottomata>	 import aggregate_projectcounts
[15:53:16] <ottomata>	 or
[15:53:21] <ottomata>	 import aggregate_projectcounts.run_git
[15:53:41] <ottomata>	 oh, or, joal, does sudo -u aggregate_projectcounts
[15:53:43] <ottomata>	 fail on push for you?
[15:53:57] <joal>	 sudo -u stats fails on commit for me
[15:54:17] <joal>	 telling me I have no user
[15:54:21] <joal>	 ottomata: --^
[15:58:54] <ottomata>	 joal: sudo -u stats git ...
[15:59:02] <ottomata>	 or sudo -u stats ../aggregate_projectcounts
[15:59:09] <ottomata>	 (or whatever path)
[16:12:16] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[16:12:56] <Ironholds>	 James_F, so if there's an SUL bug, what's the appropriate Phabricator project, do you know?
[16:13:03] <joal>	 ottomata: sorry, got to disappear for a mnute
[16:13:13] <joal>	 So when running git: warning: unable to access '/home/joal/.config/git/attributes': Permission denied
[16:13:14] <ottomata>	 ah sorry joal, i keep getting distracted
[16:13:16] <James_F>	 Ironholds: "An SUL bug"? As in a user not global? Or the concept is a mistake? Or what?
[16:13:29] <ottomata>	 yes, makes sense.  but, you can run the aggregator script itself, right?
[16:13:37] <ottomata>	 and git commands that it runs work?
[16:13:38] <ottomata>	 or not.?
[16:13:40] <joal>	 I can aggregate, but not commit
[16:13:50] <ottomata>	 as in, the git commands that the script runs fail?
[16:13:55] <joal>	 git inside agg doesn't workg
[16:14:08] <Ironholds>	 James_F, "SUL stopped working in a non-deterministic fashion"
[16:14:34] <joal>	 ottomata: this is what I'd like to run as stats: https://gist.github.com/jobar/44f4e795e484464d8581
[16:15:38] <Ironholds>	 I guess https://phabricator.wikimedia.org/project/sprint/profile/167/ ?
[16:18:17] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]
[16:18:56] <ottomata>	 yes right
[16:18:57] <ottomata>	 hm.
[16:19:12] <ottomata>	 joal: i wonder if we can make it possible for you to sudo su stats
[16:19:25] <joal>	 ottomata: I can sudo stats
[16:19:26] <joal>	  !
[16:19:38] <joal>	 But even with that, git no good
[16:20:08] <ottomata>	 right
[16:20:09] <ottomata>	 but not su
[16:20:14] <ottomata>	 you need to become stats user
[16:20:18] <ottomata>	 not just execute a command as
[16:20:22] <joal>	 correct
[16:20:33] <joal>	 I cheated like sudo -u stats bash
[16:20:38] <joal>	 But nope
[16:21:27] <ottomata>	 doesn't work, right?
[16:21:33] <ottomata>	 oh!
[16:21:34] <ottomata>	 it does!
[16:21:38] <ottomata>	 hm
[16:21:43] <ottomata>	 joal, try doing
[16:21:45] <ottomata>	 sudo -u stats bash
[16:21:54] <joal>	 II did that
[16:21:57] <ottomata>	 export HOME=/var/lib/stats
[16:22:00] <ottomata>	 then run
[16:22:04] <ottomata>	 your command
[16:22:44] <joal>	 Will tell you in 5 minutes
[16:24:14] <joal>	 ottomata: WORKED !
[16:24:19] <joal>	 Awesome :)
[16:24:40] <James_F>	 Ironholds: Yeah, not good. Thanks for filing.
[16:25:08] <joal>	 ottomata: Thanks a lot for that !
[16:25:31] <ottomata>	 cool!
[16:26:13] <Ironholds>	 James_F, np!
[16:44:00] <madhuvishy>	 ggellerman______: I'm commuting and it looks like I will be super late to retro. Feel free to start without me. I had to leave now else would not be able to make it to metrics meeting
[16:45:12] <ggellerman______>	 madhuvishy: k
[17:42:05] <madhuvishy>	 joal: https://github.com/cervisiarius/wikimedia/blob/master/navigation_trees/src/main/java/org/wikimedia/west1/traces/streaming/GroupAndFilterMapper.java
[17:44:46] <madhuvishy>	 joal: https://github.com/cervisiarius/wikimedia/blob/master/navigation_trees/src/main/java/org/wikimedia/west1/traces/GroupAndFilterMapper.java
[17:48:01] <wikibugs>	 Analytics-Backlog: Write script to track cycle time of tasked tickets  - https://phabricator.wikimedia.org/T108209#1515290 (ggellerman) NEW
[17:49:11] <wikibugs>	 Analytics-Backlog: Write script to track cycle time of tasked tickets - https://phabricator.wikimedia.org/T108209#1515315 (ggellerman) Alternatively, the end of the cycle could be when the task is marked as Resolved from Done column in the Analytics-Kanban board
[17:54:19] <wikibugs>	 Analytics-Backlog: Write script to calculate total point value of cards marked as resolved in a regular 1wk or 2 wk window - https://phabricator.wikimedia.org/T108211#1515332 (ggellerman) NEW
[18:09:15] <joal>	 milimetric: have you managed to join the meeting ?
[18:09:48] <milimetric>	 yes, joal, https://www.youtube.com/watch?v=aMQ5XVF0zm0
[18:09:56] <joal>	 Thx milimetric
[18:10:09] <joal>	 where have you found that link ?
[18:11:26] <milimetric>	 Staff channel
[18:12:01] <milimetric>	 Oops, i mean office
[18:12:24] <joal>	 K !
[18:12:27] <joal>	 Thx :)
[18:51:37] <ottomata>	 back in a bit...
[19:17:17] <wikibugs>	 Analytics-Kanban, Team-Practices-This-Week: Get regular traffic reports on TPG pages - https://phabricator.wikimedia.org/T99815#1515554 (JAufrecht) Kevin, is this done (as in automated)/should I check in in a month?
[19:43:59] <Ironholds>	 anyone remember brandon's IRC handle?
[19:45:24] <James_F>	 Ironholds: 'jorm'?
[19:45:30] <Ironholds>	 Black ;p
[19:45:38] <James_F>	 Ironholds: Oh, 'bblack'.
[19:46:05] <Ironholds>	 aha, thanks!
[19:47:15] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[19:51:16] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[19:53:16] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]
[20:17:05] <madhuvishy>	 ottomata: around?
[20:19:54] <ottomata>	 yup hiya
[20:19:56] <ottomata>	 madhuvishy:
[20:20:15] <madhuvishy>	 hey, so i made a tiny patch for the statsd stuff yesterday
[20:20:22] <madhuvishy>	 https://gerrit.wikimedia.org/r/#/c/229631/1/server/eventlogging/handlers.py
[20:21:03] <madhuvishy>	 ottomata: but i have 2 problems. 1. I'm not sure where to count actual inserted. It seems like the write place to do it is inside store_sql_events
[20:21:12] <madhuvishy>	 rigt*
[20:21:17] <madhuvishy>	 gah i cant type
[20:23:06] <madhuvishy>	 2. for some reason, if I have a url like - mysql://root:root@127.0.0.1/log?charset=utf8&statsd_host=statsd.eqiad.wmnet
[20:23:22] <ottomata>	 looking...
[20:24:02] <madhuvishy>	 argparse seems to ignore the stuff from after mysql://root:root@127.0.0.1/log?charset=utf8
[20:24:49] <madhuvishy>	 if i pass the host as first uri param it picks it up. the second one on, it strips - and I'm not sure why
[20:25:56] <ottomata>	 madhuvishy:  are you enclosing in single quotes?
[20:26:02] <ottomata>	 the whole uri?
[20:26:08] <madhuvishy>	 ottomata: no
[20:26:08] <ottomata>	 & is a special shell char
[20:26:15] <ottomata>	 will background your command
[20:26:16] <madhuvishy>	 ottomata: aah no wonder
[20:26:41] <madhuvishy>	 alright so that's not a problem then
[20:26:50] <ottomata>	 for your frist question, i'm looking at jrm.py
[20:27:00] <ottomata>	 um, do you know what insert.execute returns?
[20:27:39] <ottomata>	 hm, i am assuming _insert_multi is what we mostly use, yes?
[20:28:01] <ottomata>	 and, from that, basd on the log message that is there
[20:28:05] <ottomata>	 insert(table, events, replace)
[20:28:06] <ottomata>	             logger.info('Data inserted %d', len(events))
[20:28:16] <ottomata>	 that it is either all or none, right?
[20:28:20] <ottomata>	 if you had a batch of 100 events
[20:28:22] <ottomata>	 and 1 failed
[20:28:26] <ottomata>	 does the whole batch fail?
[20:28:59] <madhuvishy>	 ottomata: ah i don't actually know
[20:29:45] <madhuvishy>	 but yeah I considered putting it before after the log line. but I'm not sure if I should pass the statsd_host url to this
[20:30:49] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[20:31:28] <madhuvishy>	 ottomata: ^
[20:32:33] <ottomata>	 yeah i see the problem
[20:32:35] <ottomata>	 cause that's in a thread
[20:32:39] <ottomata>	 HMM
[20:33:16] <ottomata>	 madhuvishy: i think it has to be there, because the handler inserts asynchronous
[20:33:53] <ottomata>	 you could maybe give make store_sql_events take a on_success callback of some kind
[20:33:57] <ottomata>	 that you pass in
[20:34:09] <ottomata>	 something like
[20:34:21] <madhuvishy>	 ottomata: hmmm ya that could work
[20:34:31] <ottomata>	 hm, not sure what a good form of that would be
[20:34:37] <ottomata>	 hard to know how general to try to make it
[20:34:43] <madhuvishy>	 yup
[20:35:09] <madhuvishy>	 i'd also need it to return how many events it successfully inserted
[20:35:20] <ottomata>	 right, and that i don't know how you know
[20:35:25] <ottomata>	 i assume insert somehow tells you
[20:35:32] <ottomata>	 but that is sqlalchemy details maybe?
[20:35:53] <madhuvishy>	 ottomata: ah i was thinking just len(events) would do
[20:36:11] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0]
[20:36:56] <ottomata>	 i don't think so, beacause events is the batch
[20:36:58] <ottomata>	 that is attempted
[20:37:02] <milimetric>	 madhuvishy: yeah, if it attempts to insert X, it will either insert X or fail.  There's no middle state, it's in a transaction
[20:37:03] <ottomata>	 i mean, it does look like that
[20:37:08] <ottomata>	 ok
[20:37:15] <ottomata>	 so if that is the case, then you can just do len(events) madhuvishy
[20:37:19] <ottomata>	 if you can tell if insert succeeded
[20:37:26] <ottomata>	 these functions don't look like htey really return anything
[20:37:31] <milimetric>	 yep, even though it's in a batch, SQL Alchemy wraps that whole thing in a transaction
[20:37:55] <milimetric>	 you can get the number of inserted records, but I don't remember how that code looks, maybe not the way we're doing it now
[20:38:05] <milimetric>	 it's moot though
[20:38:19] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]
[20:39:23] <madhuvishy>	 ottomata: milimetric so the silliest thing to do is pass statsd_host as a param to store_sql_events - and just do stats.incr('inserted', len(events))
[20:39:48] <madhuvishy>	 it doesn't feel nice though
[20:39:51] <ottomata>	 madhuvishy: i think you should make it a callback, so as not to bring statsd dependency into the mix there
[20:39:57] <madhuvishy>	 store_sql_events shouldn't care
[20:39:58] <madhuvishy>	 yeah
[20:40:03] <milimetric>	 agree
[20:40:12] <ottomata>	 not sure what to call it
[20:40:18] <ottomata>	 on_insert_callback=None
[20:40:57] <ottomata>	 can you do partials in python ??
[20:41:03] <ottomata>	 is that what is happening all over jrm.py?
[20:41:41] <ottomata>	 ah with functools
[20:41:50] <ottomata>	 hehe
[20:41:50] <ottomata>	 https://docs.python.org/2/library/functools.html#functools.partial
[20:41:51] <madhuvishy>	 ottomata: haa
[20:41:53] <ottomata>	 could do
[20:44:37] <wikibugs>	 Analytics, Analytics-EventLogging: ServerSideAccountCreation logging captures mobile app accounts creations as "not self made" - https://phabricator.wikimedia.org/T108243#1515995 (Neil_P._Quinn_WMF) NEW
[20:44:37] <ottomata>	 increment_inserted_event = functools.partial(stats.incr, ('insertSuccessful', ))
[20:45:00] <wikibugs>	 Analytics, Analytics-EventLogging: ServerSideAccountCreation logging captures mobile app accounts creations as "not self made" - https://phabricator.wikimedia.org/T108243#1516002 (Neil_P._Quinn_WMF)
[20:45:13] <ottomata>	 PeriodicThread(target=store_sql_events, ..., args=(..., inserted_callback=increment_inserted_event)
[20:45:15] <ottomata>	 i dunno, something like that
[20:45:16] <ottomata>	 :)
[20:45:48] <ottomata>	 or maybe that's too much, not sure what the callback signature should be
[20:46:32] <madhuvishy>	 ottomata: there is also another call to store_sql_events in the finally block
[20:48:58] <ottomata>	 oh hm.
[20:49:13] <ottomata>	 yeah i guess you just have to pass the callback there too, if that's how you do it
[20:49:30] <ottomata>	 wait a minute
[20:49:39] <ottomata>	 madhuvishy:  what's that stuff about @sqlalchemy.event.listens_for
[20:49:40] <ottomata>	 ?
[20:50:01] <ottomata>	 could you potential register some event handler with sqlalchmey to do statsd stuff in???
[20:50:07] <ottomata>	 milimetric: ^
[20:50:07] <ottomata>	 ?
[20:50:43] <madhuvishy>	 but now i need to tell the callback what the statsd url is
[20:51:04] <milimetric>	 um...
[20:51:15] <milimetric>	 lemme search abit
[20:51:50] <ottomata>	 yes, that's why i was tlaking about partials!
[20:52:10] <ottomata>	 you could bind the host arg to a new partial of stats.incr :)
[20:52:21] <ottomata>	 oh sorry
[20:52:36] <ottomata>	 uhh, no, you should be able to just use stats.incr?  oh, not if that is in a thread....OR can you?
[20:52:37] <ottomata>	 i dunno.
[20:53:46] <madhuvishy>	 ottomata: ha ha
[20:54:02] <madhuvishy>	 i feel like i've forgotten all python
[20:54:54] <madhuvishy>	 ottomata: the host is for creating a StatsClient instance
[20:55:36] <ottomata>	 yes, but if you have the object, it has a function, and you could make a partial with that
[20:55:41] <madhuvishy>	 yes
[20:55:52] <milimetric>	 so, you could catch the insert event, but I'm not sure you'd want to
[20:55:53] <milimetric>	 http://docs.sqlalchemy.org/en/rel_1_0/core/events.html#sqlalchemy.events.ConnectionEvents.after_cursor_execute
[20:56:03] <milimetric>	 or higher level: http://docs.sqlalchemy.org/en/rel_1_0/core/events.html#sqlalchemy.events.ConnectionEvents.after_execute
[20:56:09] <milimetric>	 but that would be registered on the engine
[20:56:23] <milimetric>	 so if the code ever changes and we do more things with that engine, it'd be complicated
[20:56:30] <madhuvishy>	 milimetric: okay
[20:56:32] <ottomata>	 result!
[20:56:33] <milimetric>	 kind of  the same reason people don't love  triggers at the db level
[20:56:34] <ottomata>	 but result!
[20:56:39] <milimetric>	 yeah, you get the result
[20:57:15] <ottomata>	 hm
[20:57:42] <ottomata>	 doesn't MySQL have a query or maybe api method to return the last statemetn's nubmer of inserts or updates
[21:00:19] <milimetric>	 it's not an api, it returns it as a response to the batch insert
[21:00:24] <milimetric>	 it tells you how many rows were inserted
[21:00:52] <milimetric>	 why is that a problem we're solving though?  If there's no exception, it'll insert the events we passed it
[21:01:10] <milimetric>	 the result proxy we get in the listener doesn't have the insert count btw, http://docs.sqlalchemy.org/en/rel_1_0/core/connections.html#sqlalchemy.engine.ResultProxy
[21:02:02] <ottomata>	 milimetric: http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_row-count
[21:02:43] <milimetric>	 yeah, sure, but you'd have to execute that *right* after, which can't be guaranteed with concurrent threads inserting
[21:03:19] <ottomata>	 hm, they don't share the same db instance though, do they?
[21:03:21] <milimetric>	 oh i see the problem, you guys don't want to make store_sql_events smarter
[21:03:21] <ottomata>	 so probably you could
[21:03:33] <ottomata>	 well, no we will have to make it sorta smarter
[21:03:40] <ottomata>	 whatever is happening has to happen in that thread
[21:04:32] <madhuvishy>	 ottomata: i dont understand what exactly
[21:04:37] <ottomata>	 eh?
[21:04:53] <ottomata>	 milimetric: not followin gthe exception thing
[21:04:55] <ottomata>	 where would you know?
[21:05:10] <ottomata>	 store_sql_events just calls insert
[21:05:22] <milimetric>	 right, and if it throws an exception, it means it inserted 0
[21:05:28] <milimetric>	 if it doesn't, it means it inserted all of them
[21:05:44] <ottomata>	 howso?
[21:05:50] <ottomata>	 what catches that exceptoin?
[21:05:57] <ottomata>	 the thread doesn't stop, i don't see anytihng catching it
[21:06:50] <ottomata>	 i don't see an exception thrown or caught in store_sql_events, it looks like it logs logger.info('Data inserted %d', len(events)) every time, even if the insert() call failed
[21:07:40] <milimetric>	 this line: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/jrm.py#L233
[21:07:47] <milimetric>	 would never be reached, the insert above would throw
[21:07:56] <milimetric>	 so at the point it logs that, it's safe to emit to statsd
[21:07:57] <ottomata>	 woudln't the thread stop?
[21:08:06] <milimetric>	 yeah, it should die, yes
[21:08:08] <ottomata>	 ah
[21:08:18] <ottomata>	 really?  the thread dies if it encouters an error?
[21:08:48] <milimetric>	 yeah, nothing would catch the error
[21:09:20] <ottomata>	 so, woudln't that mean that all events are inserted unless there is an error that causes the consumer to die?
[21:09:20] <milimetric>	 well, so yeah, basically if the call to store_sql_events works, then the events are inserted
[21:09:48] <milimetric>	 right, so we could emit to statsd right after this line:
[21:09:48] <milimetric>	 https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/handlers.py#L279
[21:09:57] <milimetric>	 and be sure that we're telling the truth
[21:10:06] <madhuvishy>	 milimetric: oh
[21:10:19] <ottomata>	 milimetric: naw, you'd still do it in store_sql_events, because the thread is executing that
[21:10:20] <milimetric>	 it's worth testing, but I think it's as easy as that
[21:10:30] <ottomata>	 that finally only happens when the consumer dies
[21:10:33] <madhuvishy>	 ummm, i'm not sure
[21:10:40] <milimetric>	 oh duh, right
[21:10:43] <milimetric>	 that's just the cleanup call
[21:10:44] <madhuvishy>	 that is just the finally
[21:10:45] <milimetric>	 but there too
[21:10:48] <madhuvishy>	 yup
[21:10:52] <ottomata>	 ja ok, then yeah
[21:10:55] <milimetric>	 so inside store_sql_events yea
[21:10:56] <ottomata>	 just a callback to store_sql_events
[21:11:00] <madhuvishy>	 so if store_sql_events takes a callback
[21:11:02] <ottomata>	 that it will call after it calls insert
[21:11:05] <madhuvishy>	 after the log line
[21:11:06] <milimetric>	 right
[21:11:09] <madhuvishy>	 it will call it
[21:11:11] <madhuvishy>	 cool
[21:11:15] <milimetric>	 well, put the log line in the callback too
[21:11:22] <ottomata>	 meh?  whatveer :)
[21:11:24] <ottomata>	 hehe
[21:11:31] <ottomata>	 madhuvishy: add some comments about this in there whne you do this
[21:11:45] <ottomata>	 how, the thread will die and the whole process will finish if anything goes wrong
[21:11:57] <ottomata>	 so counting len(events) is correct
[21:13:03] <madhuvishy>	 ottomata: okay will do that
[21:18:49] <wikibugs>	 Analytics-Backlog: Write script to track cycle time of tasked tickets - https://phabricator.wikimedia.org/T108209#1516093 (ggellerman) Thanks to @ksmith, here is info about @Robla's script:  code: https://github.com/robla/phab-wbstatus  Running instance, pointed at some arbitrary week that contains data: http...
[22:30:23] <wikibugs>	 Analytics-Backlog, Research-and-Data, Fundraising research: What's our projected ability to fundraise in the coming years - https://phabricator.wikimedia.org/T107606#1516444 (DarTar) a:ellery
[22:44:00] <ebernhardson>	 randomly...does anyone have a bit of mysql code that will convert a mediawiki timestamp into a timestamp mysql understands and can do operations (like subtract two timestamps) on?
[22:44:13] <ebernhardson>	 the mw format is '20150806142212' or 'YYYYMMDDHHIISS'
[22:49:21] <ebernhardson>	 heh, so according to this data, the longest someone spent on a page its 3.5 days
[23:54:03] <Deskana>	 Is there a WikiMetrics web API?
[23:56:25] <Deskana>	 Let's imagine I have a web app that wants to analyse a cohort of people. Is there a way I can do that?
[23:57:03] <James_F>	 Can I mark https://phabricator.wikimedia.org/T99014 as resolved? It was…