[07:44:26] hello team! A bit of a late start this morning, need to run errand for hopefully one hour and a half [07:44:47] but if you need me I have my phone with hangouts with me :) [09:25:11] Wow, no oozie email this morning ! [09:25:21] \o/ [09:25:45] elukey: thing left this morning is to merge/deploy refinery-source+refinery [09:26:31] I think ottomata merged my stuff yesterday, double checking [09:28:45] joal: sure [09:40:50] (03PS1) 10Joal: Bump changelog to v0.0.42 before deploy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/340941 [09:40:54] elukey: --^ [09:41:42] (03CR) 10Elukey: [V: 032 C: 032] Bump changelog to v0.0.42 before deploy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/340941 (owner: 10Joal) [09:42:42] thanks elukey [09:43:02] didn't submit joal though, feel free to do it [09:43:54] !log Deploying refinery-source v0.0.42 using jenkins [09:43:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:04:46] joal: going afk again for a bit (some workers in the apt), but I'll be back soon if you need me. This morning is crazy :/ [10:04:56] Arf :( [10:05:07] elukey: let me know if there's anything I can help with [10:05:19] elukey: covering for stuff or whatever [10:05:45] joal: nono thanks! After lunch I should be fine, too many things at the same time today :) [10:06:12] elukey: with oozie mess this week, I'll be gentle this afternoon [10:09:17] (03PS1) 10Joal: Bump jar version for update spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/340945 [10:09:37] elukey: if you're around, can you give a look at --^ [10:32:07] (03CR) 10Joal: [V: 032 C: 032] "Self merging for fix-deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/340945 (owner: 10Joal) [10:32:43] !log deploying refinery [10:32:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:45:38] joal: sorry just got back :( [10:45:54] elukey: np - merged, deployed, and filled stat1002 :( [10:46:02] was waiting for you to save me [10:47:43] elukey: shall I tell scap to rollback or not? [10:47:52] elukey: I never know how to deal with a failure in scap [10:48:51] yeah you can rollback, it shouldn't be an issue [10:49:00] stat1002 is fine now [10:56:58] joal: if you want I can take care of the deploymnet [11:00:53] elukey: if you want :) [11:01:00] elukey: I can do it as well, no issue [11:01:04] elukey: you tell me :) [11:01:33] please go ahead :) [11:01:39] stat1002 is fine now [11:01:44] Doing now ! [11:02:04] !log Deploying refinery after having break stat1002 :( [11:02:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:04:21] elukey: deploy was succesfull, but smally: super fast from stat1002 - So I checked, and it's messed up :S [11:04:40] mmm how is it messed up? [11:04:45] elukey: the rev exist with the correct git log, but not all jars have been dowloaded [11:06:07] joal: what about stat1004 and an1027? [11:06:26] are they ok ? [11:06:52] stat1004 is, checking 1027 [11:07:33] the deploy was smelly in the time it took to tell me deploy was ok for canary (almost instantly, while usually it takes some time) [11:07:59] mmmm I am going to try another deploy on stat1002 only [11:08:14] an1027 ok [11:09:26] joal: can you check stat1002 again please? [11:09:31] sure elukey [11:09:46] the next step is ottomata's HAMMER [11:09:54] nope [11:10:25] * joal invokes Thor [11:15:29] joal: can you re-check now? [11:15:46] sure elukey [11:16:12] Yay, Thor always wins ! [11:16:19] Thanks a lot elukey - Sorry for the mess :( [11:16:26] \o/ [11:18:20] hi, do you know a simple url I should check to verify an sha1 exists in archiva.wikimedia.org? I'd like to automate some verifications? [11:18:35] hi dcausse - I dont :( [11:19:18] ok I'll look closer at git-fat to see how it does [11:29:31] !log Restart 3 oozie spark jobs [11:29:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:00:28] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#3070405 (10elukey) @Mpany and @Jksamra should be able to login to Pivot now (https://pivot.wikimedia.org). As I written before, please check what the `nda` LDAP group grant you access to in http... [12:02:25] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#3070414 (10elukey) As far as I can see nobody else has a pending access request, please let me know otherwise. [12:25:56] * elukey quick lunch! [12:57:31] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#3070476 (10elukey) 05Open>03Resolved Further requests will need to be a clone of this task, please open one for each interested username. Feel free to ping me directly on IRC or hangouts if y... [13:18:58] 10Analytics, 10Analytics-Cluster: Move away Hue and Camus from analytics1027 - https://phabricator.wikimedia.org/T159527#3070520 (10elukey) [13:19:04] joal: --^ [13:19:36] do yo usee any issues with that? [13:19:45] the refinery will need to be deployed on an1003 [13:21:20] 10Analytics, 10Analytics-Cluster: Move away Hue and Camus from analytics1027 - https://phabricator.wikimedia.org/T159527#3070535 (10elukey) [13:29:12] 06Analytics-Kanban, 06Operations, 10Traffic, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3070565 (10elukey) Summary before closing: Piwik was showing a lot of errors in the apache logs, we removed them and opened a task to a... [13:43:42] * elukey starting a labs instance for hadoop on debian [13:43:50] let's see what breaks :D [14:14:28] 10Analytics, 10Analytics-Cluster, 06Operations: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3070663 (10elukey) [14:27:21] (03PS1) 10Amire80: Add new error types and abuse filter details printout [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/340982 [14:31:08] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3070691 (10elukey) [14:32:41] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3070663 (10elukey) Just checked the labs instance (analytics project) cdh3-5.eqiad.wmlabs and everything seems working fine (no systemctl de... [14:33:45] ok on Monday I'll reimage analytics1040 [14:33:55] just tested in labs and we should be ok [14:34:07] moritzm: --^ \o/ [14:34:23] first step to dropping Trusty in Analytics [14:35:19] \o/ [14:39:23] what? we're not trusting analytics anymore? [14:41:49] http://giphy.com/gifs/FiCwN0LKpeP0k [14:42:05] :D [15:22:12] 10Analytics-Tech-community-metrics: Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3070802 (10Aklapper) [15:22:14] 10Analytics-Tech-community-metrics: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#3070801 (10Aklapper) [15:34:23] elukey: I don't see any problem running hue and camus on an1003 [15:34:34] elukey: I don't know what it currently does, but it's not busy :) [15:55:01] nah an camus is basically a cron [15:55:05] *and [15:55:25] running java sometimes and launching jobs :) [15:55:46] hue will run on thorium though [16:00:23] a-team: standduppp [16:00:34] joining [16:01:08] elukey: stadddupp [16:02:15] 10Analytics-Tech-community-metrics: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#3070914 (10Aklapper) p:05Lowest>03Low In [[ https://github.com/Bitergia/mediawiki-identities/commit/b856c99b5dbb02f0ccfaf48f32d5789069f2d155 | b856... [16:22:01] joal: talk about unqiues? [16:22:22] nuria: in talk with Dario and ISI - After 1-1? [16:22:39] joal: I have meetings all after 1 on 1 [16:28:07] joal:I have free time until our 1 on 1 though [16:28:25] in meeting nuria [16:28:40] impromptu meeting, but meeting nonetheless nuria [16:31:37] ok now I am really confused [16:32:01] from https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/Administration MW should send events to EventLogging first, that in turns talks with kafka [16:32:27] EventBus is frontend by an http service, so MW should not talk with Kafka for that [16:32:41] so why MW talks with kafka? :D [16:33:14] elukey: discovery uses it to send avro to kafka [16:33:17] from cirrussearch stuff [16:34:09] HHHIiiiiiiii o/ [16:34:16] you should be off! [16:34:24] i am! :) [16:34:28] thanks for the tip [16:34:38] so MW talks in IPv6 directly to Kafka [16:34:47] I was tracking down why :) [16:35:25] its done via the mediawiki monolog stuff [16:35:30] somehow [16:35:37] using a php kafka client [16:35:45] ah okok now it makes sense [16:36:24] I was talking with Faidon today and since PHP is so nice, if IPv6 for some reason fails then a bit of delay is introduced for each event sent to kafka [16:36:31] since ipv6 will be tried first [16:37:03] oh hm [16:37:25] but that's ok sorta? since the events should be sent after the request has been served back to users [16:37:32] ottomata, elukey: mediawiki api also logs with avro+kafka, for generating some information tables about api usage [16:37:39] maybe it slows down the app process handling a bit, but it shouldn't introduce request latency? [16:37:51] oh ya right, not just cirrussearch [16:38:11] ebernhardson: thanks! In fact the mw server I checked was an API one [16:38:24] ottomata: no idea, I hope that it should not [16:38:46] I was reviewing the failure scenarios if we apply the wrong IPV6 filtering to the analytics vlan :D [16:40:39] ottomata: btw I liked a lot the camus on an1003 idea [16:40:48] +1 [16:41:15] great :) [16:41:31] its not being done currently, but there are also future plans to send data from analytics back to prod over kafka at some point as well (the popularity scores we currently load directly from hadoop->elasticsearch prod, and a two way stream for sending large batches of queries from analytics to relforge that will return a bunch of feature vectors for training a machine learning ranker) [16:42:00] ebernhardson: hmMMMMM [16:42:02] we should talk about this! [16:42:13] i'm going to put a little meeting on your calendar for next week [16:42:34] ottomata: sure. the machine learning thing might finally be happening, we are looking at making an initial deployment next quarters goal [16:43:41] its not too crazy, basically some process in analytics would source a bunch of queries from logs, generate queries, push them into kafka, then something on relforge reads queries out, runs them, and sends back the feature vectors in a second channel probably [16:44:36] what is relforge? [16:45:03] 06Analytics-Kanban, 13Patch-For-Review: Replace stat1001 - https://phabricator.wikimedia.org/T149438#3071046 (10Cmjohnson) [16:45:12] it's a two server elasticsearch cluster where we run load copies of prod data (Sometimes in different ways than prod) and can run things without effecting prod [16:45:19] ahh ok [17:01:28] joal: 1 on 1? [17:35:49] going offline people! [17:35:55] o/ [17:36:00] have a good weekend [17:37:00] Bye elukey ! [17:42:22] 10Analytics, 10Analytics-Cluster, 06Operations, 10ops-eqiad, 15User-Elukey: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#2192798 (10RobH) It turns out we are out of thermal paste onsite, but I'll order more. Chris will be out for the majority of next week, b... [17:44:26] 10Analytics, 10Analytics-Cluster, 06Operations: Migrate titanium to jessie (archiva.wikimedia.org upgrade) - https://phabricator.wikimedia.org/T123725#3071405 (10Cmjohnson) [17:54:39] zareen: Hi [17:55:07] hi joal [17:55:21] I have read you etherpad and did some research [17:55:42] Computing percentile manually seems fine if it works for you [17:56:10] I wrote a request doing that [17:56:52] zareen: --^ [17:56:56] manually is fine as long as the logic works :) [17:57:11] zareen: do you have some time to double check what I did? [17:57:37] joal: yes [17:58:16] zareen: I pasted my quesry at the bottom of the etherpad [17:58:25] zareen: Can I call to discuss? [17:59:12] joal: sure, hangout? [17:59:19] yup, calling [18:23:06] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 3 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#3071515 (10Fjalapeno) Linking work in iOS here: {T158439} [18:29:14] joal: wanna talk paris - prague trains? [18:29:29] milimetric: currently with zareen [18:29:47] IRC ? I was actually thinking train from Prague to Vienna :) [18:29:50] milimetric: --^ [18:29:52] oh! [18:30:03] ok, that's simpler :) [18:30:12] Like, night train on thursday evening to get there in the morning [18:32:19] i might come to prague before the hackathon! when are y'all there? [18:34:22] Hm right, is the registration for that open [18:35:52] Nemo_bis: hackathon? yes https://lists.wikimedia.org/pipermail/wikitech-l/2017-February/087565.html [18:37:28] last time I checked the information felt incomplete [18:37:42] Ah no, now I remember, I was scared by the 12-pages form [18:37:50] * Nemo_bis delays further [18:41:32] madhuvishy: the week just before! [18:41:42] That'd be awesome to see you :D [18:44:13] (03PS6) 10Joal: Add oozie job loading MW history in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/328154 (https://phabricator.wikimedia.org/T141473) [18:45:09] (03PS9) 10Joal: Port standard metrics to reconstructed history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/322103 (owner: 10Milimetric) [18:45:44] joal: ah cool :) yeah! [18:53:18] msg madhuvishy super WELCOME To tag along, we will be there 14th to 19th, you can be with us on our working space (fits 9 people) , do you have lodging? [18:54:07] i haven't planned anything yet, but that's cool thank you! [19:23:10] (03PS1) 10Joal: [WIP] Add oozie jobs for mw history denormalized [analytics/refinery] - 10https://gerrit.wikimedia.org/r/341030 [19:33:54] Disconnecting a-team, have a good weekend ! [19:35:51] * urandom waves at joal [20:18:13] milimetric fdans map slider https://www.dropbox.com/s/ctae0wbph4lz8r2/Screenshot%202017-03-03%2015.17.50.png?dl=0 [20:18:16] make sense? [20:20:33] ashgrigas: looks good. I'd make the colors outside the selected range monochrome and maybe add some arrows on the slider handles to point towards the selected range and indicate it's interactive. Just thinking people might not be used to that kind of legend being clickable [20:24:13] milimetric what do you mean arrows? [20:24:18] havent seen that used on a slider [20:24:31] https://www.dropbox.com/s/g4w8rnhwb5uu6bi/Screenshot%202017-03-03%2015.24.28.png?dl=0 [20:24:35] you're right, people will probably get it. [20:24:37] we could also move the color part all together [20:24:45] and just be selecting the values [20:24:51] with a single fill of gray [20:25:23] no, I like that it's multi-purpose, saves space because we have to put the legend somewhere [20:25:44] no you're right, looks good like that [20:32:27] (if that was ambiguous, I meant it looks good without any arrows, but I like the colors as part of it) [22:59:41] 10Analytics: Secure hue and other private data access sites with 2FA - https://phabricator.wikimedia.org/T159584#3072318 (10Milimetric) [23:39:52] madhuvishy: the later I read your email, the more I have to read it seems. :D [23:40:01] I'm shivering as digits increase, madhuvishy. ;) [23:42:30] lzia: he he most of it is repetitive at this point [23:43:10] ok, then I read yours only, after I'm done with the current task. ;)