[00:15:40] Analytics, Operations: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2603881 (Dzahn) adding people who were on T103577 @Fjalapeno @JMinor @BGerstle-WMF as admins [00:20:02] Analytics, Operations: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2603895 (Dzahn) better source, we have the admin class "piwik-roots". 462 description: users who have root on analytics piwik servers 463 members: [@mforns, @milimetric,@nuria, @ottom... [02:38:47] (PS8) Milimetric: Script sqooping mediawiki tables into hdfs [analytics/refinery] - https://gerrit.wikimedia.org/r/306292 (https://phabricator.wikimedia.org/T141476) [07:22:39] elukey: around? [07:38:46] I am now :) [07:40:38] joal: --^ [08:11:00] elukey: I was about to book the train for sevilla [08:13:00] joal: I'd need a couple of days more because I am thinking to stay the whole week and come back on the nexy Monday.. I discovered that Ryanair offers a direct flight from Bologna but only on certain days [08:13:12] elukey: no bother :) [08:13:30] super.. you can go ahead and finish your travel arrangements, don't want to stop you :) [08:13:37] elukey: I'm going to book a train, and if you take it as well, I'll give you details :) [08:15:11] sure! thanks! [10:24:35] while working on an apache config for servermon.wikimedia.org I found a nice snippet of LDAP auth [10:25:09] we could re-use it for pivot and yarn [10:25:17] cool elukey :) [10:59:11] * elukey lunch! [11:05:59] hey morning everyone [11:06:07] I'm gonna start/finish earlier today [11:07:17] hey milimetric [11:07:23] howdy joal [11:07:23] no prob for me :) [11:07:33] not bad, good fight with scala :) [11:07:48] milimetric: do you want to go over what I have so far? [11:08:13] joal: so I finished the other patch as far as I know, addressed all of Andrew's concerns [11:08:13] milimetric: unfortunately not a lot :( [11:08:21] joal: so I finished the other patch as far as I know, addressed all of Andrew's concerns [11:08:28] milimetric: YAY ! [11:08:42] milimetric: I've seen your comment about ready for review [11:14:25] I'll have to test one more time when we sqoop simplewiki again, but after that we can merge it [11:14:35] (woah, I was just offline for like 15 minutes, IRC... psh) [11:14:42] :) [11:14:52] milimetric: awesome :) [11:15:12] joal: I have to catch up on email and phab and stuff, and I have a meeting with the ISI folks that I'd like to prepare for [11:15:22] milimetric: I'm gonna have an eye, just to understand how it works [11:15:27] but after that, do you wanna pair for like an hour before standup? [11:15:34] oh but you usually take a break around then [11:15:34] milimetric: sounds great :) [11:15:45] milimetric: I'll take my break now ;) [11:16:02] ok, cool, then we can hang out 10:00 my time [11:16:28] milimetric: booked ! [11:21:48] Taking a break now a-team [12:57:54] ehm https://github.com/github/dmca/blob/master/2016/2016-08-31-Facet.md [12:58:02] ottomata, joal --^ [12:58:28] Morning! [12:58:32] o/ [13:01:24] Hmmm uh oh... [13:01:27] milimetric: ^^^ [13:01:36] btw milimetric, did I leave my mouse at your house [13:01:39] MOUSE IN DA HOUS!? [13:01:41] yes ottomata [13:02:00] NOoOoOOo now my tennis elbow will explode! [13:02:27] whoa [13:02:27] https://github.com/implydata/pivot [13:02:41] ottomata: just take it easy today and I'll bring it tomorrow? [13:02:41] yeah [13:02:52] yeah, it's crazy they got a takedown notice [13:02:58] how shitty [13:03:12] well the reasons seems pretty compelling [13:03:20] even if I hate these kind of sw [13:03:26] yea [13:03:38] hm... man... that suxxxx [13:03:52] yeahhhh [13:03:53] yikes [13:04:02] big shame [13:04:04] hm [13:04:34] I'm sure it won't take long for them to just re-write it from scratch [13:04:50] maybe this time they'll listen to me and go with mobx / react instead of writing 100k lines :) [13:04:50] Analytics-Kanban: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2604739 (elukey) Marko gave me some reference of puppet code for a new service introduced: https://gerrit.wikimedia.org/r/#/c/305256/7 (deployed via scap) Main problem now: https://github.com/github/dmca/blob/master/2016/20... [13:07:20] milimetric: is there anything else that we can use in the meantime or we should just wait? [13:20:53] elukey: JAVA_TOOL_OPTOINS [13:20:54] OHhHhH [13:20:54] hm. [13:20:57] that is an annoying one [13:22:36] elukey: i think you suggestion is going to be the best one.. oof [13:22:42] its pretty annoying that the JVM always spits that out [13:24:11] ergh [13:25:06] elukey: we change zookeeper::server so that instead of having $cleanup_count as a parameter [13:25:10] it has $cleanup_args [13:26:12] and then we can override it from the role to include 2>&1 >/dev/null | grep -v ... [13:26:50] something like t hat [13:27:36] yeah but I would be great to keep the error messages to root@ [13:27:39] I mean, the real ones :D [13:28:01] ja but if you just grep -v out that one message from stderr [13:28:05] should work [13:28:17] ah yes I was referring to 2>&1 [13:28:25] yeah, i just got that from stackoverflow [13:28:32] ahh okok [13:28:36] apperently that will put stderr into stdout, but stdout into /dev/null [13:28:38] so |grep -v option? [13:28:47] ah wait [13:28:48] re-reading [13:28:59] elukey: or whatever incantation does it [13:29:08] there should be a way to put stderr to /dev/null [13:29:10] sorry [13:29:12] stdout to dev/null [13:29:19] and grep -v that message out of stderr [13:29:37] yes yes so atm we do that [13:29:47] we just need to | grep -v [13:29:47] we don't do the grep -v part yet [13:29:48] ja? [13:29:48] yeah [13:29:58] I believe that stdout goes to /dev/null [13:30:00] but not stderr [13:30:01] but i think you need some special incantation to /dev/null stdout and grep stderr [13:30:02] so if you grep [13:30:09] mmm [13:30:12] let me try :D [13:30:16] http://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout [13:30:19] few options in there [13:32:25] I am going to read, I was convinced that 2>&1 > /dev/null would have blackholed everything [13:33:08] Note that the sequence of I/O redirections is interpreted left-to-right, but pipes are set up before the I/O redirections are interpreted. [13:33:11] ahhh [13:33:42] woa [13:33:47] * elukey tries [13:38:16] oh my I forgot how a shell works [13:38:17] sigh [13:38:21] * elukey feels ignorant [13:40:44] haha, i mean, i just googled for that, i would have guessed the same elukey :) [13:41:56] what the hell >/dev/null 2>&1 works and 2>&1 > /dev/null doesn't... and reading the TLDP explanation it makes sense, but I was convinced otherwise [13:42:12] sigh [13:42:15] ANYHOOWW [13:42:29] I got your point now ottomata, will send a code review :D [13:43:37] pipe first, then redirect [13:43:49] * elukey still reading [13:43:53] :) [13:48:43] elukey: there's airbnb's caravel thing, or saiku if we get them to customize it for Druid [13:49:46] I mean... I hate to ask but they're clearly in trouble with Metamarkets, does that necessarily mean we can't use it? I guess so :) [13:49:58] joal: you wanna work now, my meeting ended a bit early [13:50:38] sure milimetric ! [13:50:45] milimetric: batcave? [13:50:48] omw [13:57:45] re: >/dev/null 2>&1 vs 2>&1 > /dev/null - not only a pipe vs redirection priority, but simply a dup() issue.. duplicating fd before or after DOES make the difference [13:57:56] really weird that I've stumbled in this issue only now [14:00:48] elukey: i kinda like the bash suggestion...not sure if crontab is bash (probably not?) [14:01:03] command 2> >(grep 'something') >/dev/null [14:01:07] redirect into a subshell [14:01:55] oh or [14:01:55] command >/dev/null |& grep "something" [14:01:56] ? [14:07:48] in the cron we should have atm something like > 1 > /dev/null to isolate only stderr no? [14:07:54] so | grep -v should suffice [14:07:55] IIRC [14:08:49] yeah that sounds fine [14:08:57] many ways to do it, i'm not picky [14:09:06] we just want to make sure we don't force that on the zk module [14:09:20] and just change the param to $cleanup_args and default it to what ends up being passed now [14:10:59] sure [14:11:08] and then we'll customize it case by case [14:25:41] ottomata: https://gerrit.wikimedia.org/r/#/c/308173/2 ? [14:26:24] hm, elukey sounds fine [14:26:29] i was thikning of getting rid of $cleanup_count [14:26:31] and making [14:26:35] $cleanup_script_args default to [14:26:39] -n 10 >/dev/null [14:26:47] ahhh okok [14:26:49] oh but [14:26:50] hmmm [14:26:57] I think it is ok [14:26:57] $cleanup_count is used to conditionally install thecron [14:27:06] wait, is it? [14:27:06] mmmm [14:27:08] the docs says it is! [14:27:38] but afaiu from puppet it does not.. [14:27:39] hm! wonder when that changed. [14:27:39] yeah [14:27:41] looks like not [14:27:54] that sounds useful... [14:28:14] # if !$cleanup_count, then ensure this cron is absent. [14:28:14] if (!$cleanup_count or $cleanup_count <= 0) { [14:28:14] Cron['zookeeper-cleanup'] { ensure => 'absent' } [14:28:25] do it with ensure [14:28:57] $cleanup_cron_ensure = ? $cleanup_count { [14:28:58] 0 => false, [14:28:58] default => true [14:28:58] } [14:28:58] [14:29:02] ensure => $cleanup_cron_ensure [14:29:17] or maybe also false => false [14:29:18] just in case :) [14:29:26] haha [14:29:33] but if we put a if cleanup_count > 0 before the cron's declaration? [14:29:47] puppet won't clean it up if it is changed [14:29:53] it needs to be done via the cron's ensure param [14:29:56] ensure [14:29:59] sorry, not ensure true [14:30:13] ah it will work only for fresh installs [14:30:14] absent/present [14:30:15] yes [14:30:44] so, contitionally set ensure to either present or absent based on the value of $cleanup_count [14:33:13] I don't get what is wrong with the current code though [14:34:01] that doesn't ensure in case > 0? [14:34:23] afaik the cron {..} should do it no? [14:35:12] maybe we could do something like this [14:35:25] if cleanup_script is defined then we ensure the cron [14:35:30] otherwise we don't [14:35:42] and -n could go in cleanup_count [14:35:54] sorry cleanup_script_args [14:41:03] put it in https://gerrit.wikimedia.org/r/#/c/308173/3/manifests/server.pp [14:47:31] https://puppet-compiler.wmflabs.org/3923/ looks good [14:47:37] anyhow, this is only a proposal [14:47:41] let me know if you like ti [14:47:43] *it [15:09:07] (CR) Ottomata: [C: 1] "Only one nit! Other than that +1" (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/306292 (https://phabricator.wikimedia.org/T141476) (owner: Milimetric) [15:10:10] (CR) Milimetric: Script sqooping mediawiki tables into hdfs (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/306292 (https://phabricator.wikimedia.org/T141476) (owner: Milimetric) [15:21:23] elukey: 2 things [15:21:39] 1 set the ensrue variiable before you define the cron [15:21:41] and just set [15:22:01] ensure => $cleanup_script_cron [15:22:23] there's no reason to use the Cron override thing there [15:22:56] less certainly, i feel like using $cleanup_script or disabling the cron is hackier than using cleanup_count [15:23:00] maybe cleanup_count is hacky too [15:23:29] I find it clear.. if you define the script it will be deployed, otherwise it won't [15:23:31] but, if you think so, then maybe we should just stop overloading the meaning of that variable and make $cleanup_enabled parameter [15:23:57] could be an option yeah [15:24:00] more direct [15:24:02] and explicit [15:24:12] hmm, i dunno, i think i see $cleanup_script more simliar to the $...template params i often use [15:24:19] its more for overriding in case you have a really special use case [15:25:16] I didn't get this one sorry :D [15:25:23] well, ok so thing slike [15:25:26] $log4j_template [15:27:10] its parameterized in case the log4j_template that the module ships does not work for you [15:27:15] maybe you need more complicated log4j stuff [15:27:24] this way you can provide your own to the module and it will be used instead [15:27:37] same kinda thing for cleanup script...although maybe i shouldn't have parameterized it in the first place [15:27:44] i was thinking in case you used a different zk package [15:27:52] the script might exist in a different location, or as a different name [15:28:04] but, that is really unlikely and maybe kinda silly to worry about [15:28:38] yeah makes sense, but maybe you don't want the cron for some reason [15:29:06] yeah, that should be able to be disabled, but i don't think doing it via the $cleanup_script param is good, it works but it feels wrong to me :p [15:29:20] elukey: let's make us both happy and add the $cleanup_enabled parameter [15:29:27] i'd be fine with removing $cleanup_script btw [15:29:33] and just hardcoding that one [15:38:52] all right https://gerrit.wikimedia.org/r/#/c/308173 [15:49:13] elukey: almost [15:49:14] just do [15:49:20] ensure => $cleanup_cron_ensure [15:49:27] directly on the cron declaration [15:49:34] no need for the extra conditional [15:49:38] ah snap I wanted to do it but I forgot [15:49:42] * elukey blames fridays [15:51:04] ok done ottomata :) [15:51:42] elukey: +1, merge at will [15:51:43] :) [15:52:14] goood :) will do it on Monday [16:05:10] milimetric: an hour of pair coding? [16:14:28] joal: sorry i'm in an interview [16:14:32] after? [16:14:35] np milimetric [16:14:37] but is ok if you're gone' [16:14:52] I'll work until end-of hour [16:20:39] team going afk! Have a good weekend [16:20:42] :) [16:20:43] o/ [16:51:52] joal: quick finish of the day chat? [16:52:05] yessir ! [16:52:28] I'm in da cave [17:01:40] milimetric: sorry, clicked too fast !!! [17:01:45] milimetric: have a good weekend :) [17:01:50] you too [17:02:33] if you need to sqoop again, you should be able to use the python in my latest patch, but I'm thinking the data that's already there is fine [17:02:36] to test [17:02:46] gonna go get lunch, see yall [17:03:17] Going off a-team, have a good weekend :) [18:15:13] (PS1) MaxSem: Always read config from script directory [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/308211 [18:21:03] milimetric: I think i have fixed the bug i had with dateranges and graph but I think it was there before, we were just not seeing it for some reason that i suspect is due to how binding is being throttled now. will send codepatch and explain [18:22:18] (CR) Yurik: [C: 2 V: 2] Always read config from script directory [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/308211 (owner: MaxSem) [18:22:51] (PS1) MaxSem: Always read config from script directory [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/308214 [18:22:59] (CR) MaxSem: [C: 2 V: 2] Always read config from script directory [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/308214 (owner: MaxSem) [18:38:49] let me know when to take a look nuria_ [19:56:48] milimetric: check this out: http://druid.io/ [19:57:09] milimetric: metamarkets links to that page on his job page to talk about open source [19:57:23] and in turn that links to pivot [19:57:35] which they have taken down [19:57:48] milimetric: https://metamarkets.com/jobs/ [19:58:01] ha [19:58:54] (PS8) Nuria: Bookmark for browser dashboard regarding graph and time [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [20:00:24] milimetric: ok, i think i have ironed issues with bootstrap of initial state from bookmark, needed to convert from ISO date format to unix timestamp as it is what timeseries data uses [20:01:16] milimetric: I have also reduced the amount of events we listen for in the datepicker, thus (hopefully) reducing KO updates: https://gerrit.wikimedia.org/r/#/c/306980/8/src/app/ko-extensions/datepicker-binding.js [20:02:06] cool, I'll check out the patch now but I haven't looked closely at this yet [20:04:25] milimetric: ya, no rush at all but the patch that marcel CR-ed had a bug with bootstrapping the initial state, everything else worked i think. I just wished the code would be less inter-twined with component code. [20:04:45] milimetric: It can be, i just need to do a bigger refactor but maybe now it is not the time [20:05:56] Hallo. [20:06:01] we should think on it, but I think now's the best time to fix up dashiki. Before we have to lean on it more and more [20:06:26] Does anybody have an idea why are there much more pageviews on tl.wikipedia (Tagalot) since mid-June? [20:06:27] https://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&start=2015-07-01&end=2016-09-01&sites=tl.wikipedia.org [20:06:38] s/Tagalot/Tagalog/ [20:10:46] no idea, aharoni, but could be the Chrome 41 bug or simple spikes in news, those affect smaller wikis much more [20:13:17] milimetric: I think we should do state management + history support together but there is no scaping the fact that there is no magic and some crafty code needs to be written [20:14:34] aharoni: sudden increments are due to either: non-identified-crawlers or for very small wikis sudden bursts associated to a page that went viral [20:15:05] aharoni: you have access to the data so you can see all pages viewed and see whether there are spikes on 1 page for example [20:16:02] aharoni: spike is pretty huge: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=tlwiki/metrics=Pageviews [20:16:26] Analytics, Pageviews-API: Much more pageviews in Tagalog Wikipedia since mid-June 2016 - https://phabricator.wikimedia.org/T144635#2605825 (Amire80) [20:16:39] aharoni: but it does not correlate with chrome 41 bug [20:19:05] https://www.irccloud.com/pastebin/RGKpalCj/ [20:19:24] aharoni: on june 12th a new version of mw was deployed [20:20:10] nuria_: there is another similarity to the Main Page issue ( https://phabricator.wikimedia.org/T141506 ) [20:20:34] In Tagalog, too, it's the Main Page that started getting much more views. [20:22:14] aharoni: no, i think the "main page" issue also is affecting the data but unlike other wikis traffic doesn't go down [20:23:03] aharoni: compare ruwiki and tagalog [20:23:05] https://analytics.wikimedia.org/dashboards/vital-signs/#projects=tlwiki,ruwiki/metrics=Pageviews [20:32:38] Analytics, Pageviews-API: Much more pageviews in Tagalog Wikipedia since mid-June 2016 - https://phabricator.wikimedia.org/T144635#2605825 (Nuria) @Amire80 Issue doesn't look seasonal: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=tlwiki/metrics=Pageviews Neither per se related to i... [20:38:58] Analytics, EventBus, Services: Check eventbus Kafka cluster settings for reliability - https://phabricator.wikimedia.org/T144637#2605897 (GWicke) [20:39:49] Analytics, EventBus, Services: Check eventbus Kafka cluster settings for reliability - https://phabricator.wikimedia.org/T144637#2605884 (GWicke) [21:06:43] (CR) Milimetric: "This is ok, but it adds tech debt that I think needs to be addressed right away. So if we want to get these features out right away, then" (5 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) (owner: Nuria) [21:07:05] ok nuria_, I wrote down some comments inline, and we can chat about it if you want [21:07:14] I'm about to sign off for the days soon, but I have like half hour [21:08:18] Analytics, Reading-analysis, Research-and-Data, Research-consulting: [Epic] Update official Wikimedia press kit with accurate numbers - https://phabricator.wikimedia.org/T117221#2605943 (leila) [21:13:53] Analytics, Reading-analysis, Research-and-Data, Research-consulting: Propose metrics along with qualifiers for the press kit - https://phabricator.wikimedia.org/T144639#2605965 (leila) [21:14:36] Analytics, Operations: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2605979 (Milimetric) I was logged in and it looks like my session is still good. But if I try to login with the credentials for piwik-wmf-admin, it fails authentication. Only two possibilities... [21:14:57] Analytics, Reading-analysis, Research-and-Data, Research-consulting: [Epic] Update official Wikimedia press kit with accurate numbers - https://phabricator.wikimedia.org/T117221#1769033 (leila) [22:52:38] Analytics, Pageviews-API: Much more pageviews in Tagalog Wikipedia since mid-June 2016 - https://phabricator.wikimedia.org/T144635#2606180 (Tbayer) >>! In T144635#2605867, @Nuria wrote: > @Amire80 > > Issue doesn't look seasonal: > https://analytics.wikimedia.org/dashboards/vital-signs/#projects=tlwiki...