[02:59:37] <grrrit-wm>	 (PS22) Csalvia: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[02:59:39] <grrrit-wm>	 (PS10) Csalvia: Add scheduler mode to wikimetrics [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/111623 (owner: Milimetric)
[02:59:41] <grrrit-wm>	 (PS9) Csalvia: Allow reports to be rerun [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/111914 (owner: Milimetric)
[02:59:43] <grrrit-wm>	 (PS1) Csalvia: Add concatenated recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119693
[03:00:07] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Add concatenated recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119693 (owner: Csalvia)
[08:40:14] <grrrit-wm>	 (CR) QChris: "Since pom.xml declares the repository as GPLv2 (not GPLv2+) and Apache" [analytics/kraken] - https://gerrit.wikimedia.org/r/119301 (owner: Ottomata)
[08:45:43] <grrrit-wm>	 (PS1) QChris: Remove Java part of kraken-etl [analytics/kraken] - https://gerrit.wikimedia.org/r/119724
[13:00:21] <grrrit-wm>	 (CR) Milimetric: Add concatenated recurrent reports (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119693 (owner: Csalvia)
[13:26:03] <grrrit-wm>	 (PS1) QChris: Force provision when reloading vagrant [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119742
[13:51:35] <grrrit-wm>	 (CR) Ottomata: "Gah, I just tried to put a license on them that I was sorta familiar with. I didn't see a main license file in Kraken so I just picked on" [analytics/kraken] - https://gerrit.wikimedia.org/r/119301 (owner: Ottomata)
[13:53:22] <grrrit-wm>	 (CR) Nuria: [WIP] Run recurring reports using the scheduler (16 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[13:58:15] <qchris>	 We're trying appear.in today aren't we?
[13:58:18] <qchris>	 https://appear.in/wmf-analytics-batcave
[14:06:22] <matanya>	 nuria: dvanliere isn;t with wmf anymore?
[14:17:46] <ottomata>	 matanya: that is correct
[14:18:04] <matanya>	 ottomata: i'll remove him from icinga then
[14:18:26] <ottomata>	 k danke
[14:26:42] <matanya>	 ottomata: if you give me the names of the new analytics members i'll add them too
[16:48:08] <rdwrer>	 milimetric: How do the automatic graph generators work? Is it possible to edit them, or do I need to start over if I want to change specific things?
[16:50:33] <Ironholds>	 rdwrer, they're in coffeescript, so we don't actually know.
[16:50:44] <Ironholds>	 We just splash the sanctified chicken blood on the right altars and hope.
[16:50:58] <Ironholds>	 which is pretty much the process that led to someone coming up with CS in the first place.
[16:51:20] <rdwrer>	 You...hrm
[17:49:19] <milimetric>	 rdwrer / Ironholds: sorry I've been in meetings Literally all day so far
[17:49:25] <milimetric>	 I should be done soon and I'll help
[17:49:32] <milimetric>	 oh, sorry terrrydactyl as well, sorry!
[17:49:49] <rdwrer>	 'kay
[17:49:52] <terrrydactyl>	 no rush
[17:50:05] <Ironholds>	 milimetric, heh, it's fine :0
[18:36:22] <halfak>	 Hey ottomata.  Let me know when you have a few minutes to discuss https://rt.wikimedia.org/Ticket/Display.html?id=7090
[18:37:02] <ottomata>	 sure
[18:37:22] <ottomata>	 do we have slave lag alerts in general already set up elsewhere?
[18:37:52] <halfak>	 I'm not sure.
[18:38:02] <halfak>	 I've seen springle ping due to lag before.
[18:42:37] <ottomata>	 totally
[18:42:37] <ottomata>	 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db1047&service=MySQL+Slave+Delay
[18:42:52] <ottomata>	 haha, notifications disabled on that one :p
[18:43:21] <halfak>	 Heh.  Should probably stay disabled for everyone except us lag causers
[18:43:48] <halfak>	 ottomata, what credentials does icinga.wikimedia.org want?
[18:43:59] <halfak>	 n/m
[18:44:03] <halfak>	 I should read
[18:45:01] <ottomata>	 L)
[18:45:02] <ottomata>	 :)
[18:45:16] <halfak>	 Hmm... Doesn't like my labs wiki login.  Maybe I have to be added to a project?
[18:45:38] * halfak  just logged in at wikitech to be sure

[18:47:47] <ottomata>	 ok, i can make this happen
[18:47:53] <ottomata>	 yaeh maybe, i'm not sure icinga is open
[18:47:54] <ottomata>	 dunno
[18:48:02] <ottomata>	 so, halfak, i guess I should make it notify the analytics contact group?
[18:48:08] <ottomata>	 the one that daniel just updated?
[18:48:58] <halfak>	 Just so long as you guys don't mind the pings.  Otherwise, I'd opt to just ping those us of us who cause and are affected by the lag.
[18:49:27] * halfak  is not 100% positive that researchers/analysts are the only ones affected by lag. 

[18:50:30] <ottomata>	 well we'd need to set up a separate research contact group for you guys to do that
[18:50:34] <ottomata>	 sooo ok analytics it is :)
[18:50:43] <halfak>	 :)
[18:51:22] <ottomata>	 hmm, ok this is going to make is so that any mysql alert for those nodes will get to us
[18:51:23] <ottomata>	 soooo ok!
[18:51:36] <halfak>	 Thanks for your help.
[19:03:07] <ottomata>	 halfak: https://gerrit.wikimedia.org/r/#/c/119797/
[19:03:58] <halfak>	 Saw that.  Anything for me to except take notice?
[19:04:04] <ottomata>	 naw
[19:04:05] <halfak>	 ottomata, ^
[19:04:08] <halfak>	 cool :)
[20:41:01] <grrrit-wm>	 (PS1) Ottomata: maven-shade-plugin generates this file and changed it for Hadoop 2 [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/119868
[20:44:37] <grrrit-wm>	 (PS23) Nuria: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[20:49:03] <grrrit-wm>	 (PS24) Nuria: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[20:52:32] <grrrit-wm>	 (CR) Ottomata: "Actually, I just submitted this upstream." [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/119868 (owner: Ottomata)
[21:05:18] <yurik>	 ottomata, do you know how logging works with varnish? at which point does the log string is formulated - after vcl_deliver?
[21:08:01] <ottomata>	 yurik, probably, it is forumlated as a separate process, currently varnishncsa (and also sometimes varnishkafka)
[21:08:08] <ottomata>	 varnish keeps shared memory logs of all requests
[21:08:21] <ottomata>	 separate processes read those logs and then create string logs from them
[21:08:31] <ottomata>	 and send them either via udp (varnishncsa) or kafka (varnishkafka)
[21:08:44] <yurik>	 right, but at which point does it actually formats the log string?
[21:08:52] <ottomata>	 but, I do not know at what point the log gets stored in the shared memory logs
[21:09:04] <ottomata>	 its formatted by varnishncsa
[21:09:09] <yurik>	 thx
[21:09:13] <yurik>	 will have to explore that
[21:09:17] <ottomata>	 which is a completely standalone process from varnish
[21:09:31] <ottomata>	 so, if you can figure out when varnish saves a log in its shared memory logs
[21:09:38] <ottomata>	 then you will know what data is there
[21:09:49] <yurik>	 still it has to be in sync with varnish - you can't format log string before it is done.
[21:09:59] <grrrit-wm>	 (PS2) Terrrydactyl: WIP Removing cohorts from database [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343
[21:10:02] <yurik>	 guess its after vcl_deliver, this way i can still alter some of the values  in it
[21:10:04] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] WIP Removing cohorts from database [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 (owner: Terrrydactyl)
[21:10:35] <yurik>	 thx
[21:13:30] <grrrit-wm>	 (PS1) QChris: Move kraken-etl from Apache License 2.0 to GPL-2+ [analytics/kraken] - https://gerrit.wikimedia.org/r/119874
[21:14:07] <grrrit-wm>	 (CR) QChris: "> We can(?) change it to GPL?" [analytics/kraken] - https://gerrit.wikimedia.org/r/119301 (owner: Ottomata)
[21:16:16] <ottomata>	 thanks qchris, I don't think they'll care
[21:16:24] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Move kraken-etl from Apache License 2.0 to GPL-2+ [analytics/kraken] - https://gerrit.wikimedia.org/r/119874 (owner: QChris)
[21:16:48] <qchris>	 Licenses suck :-(
[21:18:08] <grrrit-wm>	 (CR) QChris: "https://gerrit.wikimedia.org/r/#/c/119874/" [analytics/kraken] - https://gerrit.wikimedia.org/r/119301 (owner: Ottomata)
[22:07:16] <milimetric>	 rdwrer: I just got free :)
[22:07:24] * rdwrer  checks watch

[22:07:27] <milimetric>	 if you're still around I'd love to answer your question finally
[22:07:30] <rdwrer>	 Sure!
[22:07:33] <milimetric>	 k
[22:07:35] <rdwrer>	 milimetric: !answer :)
[22:07:44] <milimetric>	 milimetric: How do the automatic graph generators work? Is it possible to edit them, or do I need to start over if I want to change specific things?
[22:08:25] <milimetric>	 automatic graph generator takes the URL, reads it as if it was the defined URL for a datasource, infers the format (json/tsv/csv) and columns
[22:08:41] <milimetric>	 then it makes a graph with a line for each column it finds
[22:08:47] <milimetric>	 all of this client side
[22:08:54] <milimetric>	 and it generates no artifacts
[22:09:00] <milimetric>	 so it doesn't leave behind anything for you to change
[22:09:20] <milimetric>	 now - if you'd like to customize some part of how it does this, we could extend the trick I used to pass options right from the dashboard
[22:09:36] <milimetric>	 but in practice, ou
[22:09:59] <milimetric>	 *in practice, you'd have to have a pretty defined set of requirements up front and I could probably only do this during my free time
[22:10:22] <milimetric>	 OR! I could teach you coco (i'm fairly serious, it's not too hard, especially isolated to this change)
[22:11:13] <milimetric>	 so - the question is, rdwrer, what would you like to change about the way it generates the graphs?
[22:12:08] <rdwrer>	 milimetric: I'd like to set human-readable labels for the various lines
[22:12:20] <rdwrer>	 Fabrice has complained that "image-mean" is not easy to understand
[22:12:24] <milimetric>	 oh, totally
[22:12:42] <milimetric>	 so, that's just what comes out of the scripts that create your datafile
[22:12:55] <milimetric>	 just change it to select blah as 'happy column name'
[22:13:20] <rdwrer>	 Ugh, 'kay
[22:13:32] <rdwrer>	 Having spaces in identifiers skeeves me out
[22:13:35] <rdwrer>	 But if I must...
[22:13:54] <milimetric>	 oh those aren't identifiers, sql doesn't do anything with them unless you're using it as a subquery
[22:14:30] <milimetric>	 and even so, sql is totally happy dealing with spaces in column names, that's what [] and '' are for
[22:14:49] <milimetric>	 also - the alternative is creating the graph and datasource files and changing those as much as you want
[22:29:40] <grrrit-wm>	 (CR) QChris: "Also, it seems the short-cut for debug mode creates a hard" (6 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/111623 (owner: Milimetric)
[22:41:21] <milimetric>	 ottomata: I think the license thing can only be solved by taking those files out into a separate repository
[22:41:38] <ottomata>	 is there a reason kraken is GPL-2?
[22:41:39] <milimetric>	 maybe like "wmf-python-utils"?
[22:41:40] <milimetric>	 ugh
[22:42:06] <milimetric>	 well, you said it was because the POM files specify it
[22:42:20] <milimetric>	 and I assume they do because they use other software that's GPL 2
[22:42:28] <milimetric>	 as nobody would randomly pick that license - it's the evil one
[22:43:52] <milimetric>	 ugh was to that ugly name for the repository
[22:44:15] <milimetric>	 I'm really not sure what best practices are for super useful util functions...
[22:46:40] <ottomata>	 i think it should be in kraken, let's not change that right now
[22:46:45] <ottomata>	 GPL-2 is the evil one?
[22:46:53] <ottomata>	 i dunno, maybe david just picked it?
[22:52:00] <milimetric>	 i *highly* doubt david just picked that :)
[22:53:55] <milimetric>	 if anyone just picked it, it'd be like the hardcore Linus Torvalds type
[22:55:59] <ottomata>	 hmm, i so much don't care about licenses.  Can I just write at the top of the python file "DO WHATEVER YOU WANT I DON"T CARE"
[22:56:01] <ottomata>	 haha
[22:56:03] <ottomata>	 ok i gotta run
[22:56:03] <ottomata>	 polo time
[22:56:07] <ottomata>	 laaaaaters!
[23:02:52] <Ironholds>	 'do whatever you want' ah, the MIT license.
[23:03:02] <Ironholds>	 GPL2 isn't evil. And neither is smallpox.
[23:03:06] <Ironholds>	 for the same reasons
[23:09:46] <halfak>	 +1 MIT licenses.
[23:10:46] <halfak>	 Ironholds, there's also https://github.com/nslater/DAMAIL
[23:10:57] <halfak>	 "Copying and distribution of this file, with or without modification, are permitted in any medium provided you do not contact the author about the file or any problems you are having with the file."
[23:11:09] <Ironholds>	 halfak, and don't forget http://www.wtfpl.net/
[23:11:31] <Ironholds>	  DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
[23:11:31] <Ironholds>	    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
[23:11:31] <Ironholds>	   0. You just DO WHAT THE FUCK YOU WANT TO.
[23:12:04] <halfak>	 Heh.  Except you have to change the name when you make copies.
[23:12:43] <Ironholds>	 s'true
[23:12:50] <milimetric>	 WTF is my favorite license
[23:13:35] <milimetric>	 as I just said in an email, restrictive open source licensing is like idealist on idealist crime - hurts everyone
[23:17:18] <Ironholds>	 milimetric, yeah. I go MIT by default for code and CC-BY-SA or CC-0 for non-code
[23:17:29] <Ironholds>	 (CC-0 if I get a choice, CC-BY-SA if it's WP. Or my hand.)