[08:06:23] http://techblog.netflix.com/2016/02/evolution-of-netflix-data-pipeline.html [08:06:26] :) [08:29:38] joal, goooood morning! Let me know if you are free today to start the Kafka brokers reboot [08:33:12] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} [8 pts] - https://phabricator.wikimedia.org/T124676#2034561 (jcrespo) There is a problem I didn't reali... [09:08:22] Morning elukey ! [09:08:26] I'm all for you :) [09:09:47] hellooooo [09:10:03] I am wondering if we can test a reboot in beta first.. [09:10:13] we should have a kafka cluster in there right? [09:10:40] elukey: We have a kafka cluster I think, but no hadoop yet IIRC [09:10:46] But we could do with EL [09:11:58] sorry joal I meant to check if mediawiki breaks [09:12:05] it shouldn't since we have the new patch [09:12:06] k elukey :) [09:12:07] but.. [09:12:32] I don't think there is the equivalent MW-Kafka link en beta ... Maybe I'm wrong [09:13:17] yeah probably, checking in labs now [09:13:30] moreover nobody re-added kafka1012 to mediawiki-config yet [09:14:12] elukey: not sure that last info is really helkpful :) [09:15:15] joal: do you mean labs?? [09:16:04] no, the kafka1012 not being added [09:16:42] For me, not having 1012 in the list means that varnish will send to less brokers (particularly when rebooting them) [09:17:35] so the config that I am talking about is for the things that mediawiki pushes to kafka (discovery?) [09:18:16] Ok right, I thought it was varnish related, my bad :) [09:18:27] I'm gonna drink some more coffe and try to concentrate :) [09:18:33] nono sorry I didn't explain myself correclty, apologies [09:19:00] So, volume from MW(discovery) is relatively small (compared to webrequests) [09:19:17] It could be ok to reboot even if kafka1012 is not in the broker list [09:19:23] super [09:19:24] kafka300 login: [480173.541160] Out of memory: Kill process 26212 (java) score 399 or sacrifice child [09:19:28] this is labs [09:19:31] in the console [09:19:32] :D [09:19:32] elukey: Or at least, I think it could :) [09:19:55] looks like labs kafka is a bit overloaded :) [09:20:10] now I remember Andrew talking about beta/labs brokers broken :D [09:22:18] I guess we are gonna have to make it live [09:23:22] yep probably [09:26:08] all right, going to inform ops and then I'll proceed in this way: 1) check partions status and replication 2) kafka service stop on kafka1013 3) reboot [09:31:25] elukey: ok, after reboot, monitor partition status & replication before proceeding again :) [09:32:33] yep I am talking with ops about depooling first from mediawiki, just to be super sure [09:33:27] elukey: great :) [10:30:27] elukey: Will you need me in the next 2 hours or can I take a leave ? [10:35:54] elukey: no answer, I leave :) [10:36:04] elukey: I'll be back in less than 2 hours [10:37:50] joal: sorry just seent the message, you can go :) [11:46:49] elukey: I'm back ! [11:56:20] joal: back too! [11:59:09] so the plan is to prepare the removal of kafka1013 from mediawiki-config but not to sync it. In case of turbolence we'll be able to fix the issue straight away, and at the same time we'll test in prod the last hhvm fix [12:03:52] Sounds good elukey :) [12:04:16] elukey: freenode has issues, let's sync on hangout [12:05:21] freenode is rebooting today ... they got that announced earlier [12:06:26] Just noticed the notice, thx elukey for the pointer :) [12:08:59] back :D [12:26:30] halfak: Hello ! Must be very early for you [12:26:52] halfak: Can we schedule some time either today or tomorrow for a smalltalk on sanitization? [12:45:09] joal: still preparing the change, sorry for the delay :) [12:45:19] np elukey, no rush is good :) [12:56:01] joal: stopping kafka on 1013, partitions looks good [13:08:01] !log stopped kafka on kafka1013 for maintenance [13:08:08] joal --^ [13:11:30] elukey: Seen! [13:13:16] elukey: camus handles fine [13:14:05] elukey: seems that EL doesdn't ... [13:17:16] joal: :( [13:17:36] If you want I can bring the service up again [13:17:41] to figure out what's happened [13:18:27] elukey: I'll restart EL now [13:19:31] all right! [13:20:30] !log Restarting EventLogging after kafka-restart-related outage [13:20:59] joal: let me know when I can proceed with the reboot [13:21:32] elukey: I think there is no more issue now that it restareted and kafka1013 is not up [13:21:50] all right! [13:27:25] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [30.0] [13:28:28] (Abandoned) Mforns: Improve the data format of the browser report [analytics/refinery] - https://gerrit.wikimedia.org/r/270955 (https://phabricator.wikimedia.org/T126282) (owner: Mforns) [13:28:53] joal: kafka1013 is up and running again [13:29:21] k elukey, reviewing camus and el [13:29:34] camus seems fine [13:31:14] EL is weird [13:34:07] joal: just arrived from burrow [13:34:08] eventlogging-client-side:3 (1455713704500, 187577488, 0) -> (1455715411909, 187591078, 0) [13:36:08] elukey: Will restart again :S [13:44:44] joal: here I am sorry, was finishing in ops [13:44:56] anything useful from the logs? [13:47:31] elukey: problem with async producing when metadata change from whatI see [13:47:41] restart has solved the thing [13:48:01] chart looks good now [13:48:47] seems to mean that restart is need at watever metadata change there is [13:49:09] Because last you made didn't involve partition leader reelection, correct? [13:49:20] elukey: --^ [13:49:21] nope, I didn't run it [13:49:29] right [13:49:53] So restart is needed when kafka goes down, AND when it comes back :( [14:03:54] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [14:09:50] (PS1) Mforns: Improve the format of the browser report [analytics/refinery] - https://gerrit.wikimedia.org/r/271253 (https://phabricator.wikimedia.org/T126282) [14:17:48] a-team: something to follow: https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces87 [14:18:11] elukey: shall we proceed with other kafka brokers ? [14:18:37] ooh, Arrow got promoted [14:18:45] :) [14:19:26] I'd like to understand more of arrow / parquet diffs [14:21:23] joal: still talking in security about next steps etc.., I'll ping you straight away for the next step [14:21:44] perfect elukey, how did MW handle the reboot ? [14:25:01] super fine! [14:25:17] great :) [14:25:35] Our only issue is EL then (camus went fine as well) [14:26:07] hiyaaa [14:26:16] ottomata Helllllo [14:26:22] saw that el page, and remembered that we haven't yet upgraded pykafka :/ [14:26:35] Arff ! [14:26:47] I thought it was done (I should have double checked :( [14:32:16] maybe i'll take a moment and build the .deb right now... [14:32:42] ottomata: If you have time to do it before next kafka restart that's be awesome :) [14:32:48] Analytics-EventLogging, Analytics-Kanban: Upgrade pykafka to v2.2 - https://phabricator.wikimedia.org/T126075#2035179 (Ottomata) a:Ottomata [14:32:54] how far along are we? [14:35:04] ottomata: hello! [14:35:46] I rebooted kafka1013 only and discussing in security how to proceed to make sure that mediawiki won't do weird things. HHVM worked perfectly with the new patch, good news :) [14:36:53] ok great [14:36:57] working on new deb now [14:54:07] joal: http://devopsreactions.tumblr.com/post/139476901765/trying-to-identify-a-problem-without-adequate [14:54:27] :D elukey [14:54:49] feels like me with EL :) [14:54:59] (not even permission related) [14:55:00] :D [14:55:21] I can picture you saying "MEAOOWW" while debugging [14:55:28] :D [14:55:39] Would be a grumpy MEOOOW :) [15:08:14] elukey: i have built debws [15:08:15] debs [15:08:21] lemme upgrade eventlog1001 and restart EL there [15:08:25] super [15:08:32] thanks for the detailed explanation in security [15:08:40] :) [15:12:05] !log restarting eventlogging with pykafka 2.2. [15:12:06] 0 [15:12:54] \0/ [15:13:59] elukey: nope [15:14:00] no good [15:14:04] that's what I get for not testing it :) [15:14:20] some gevent dependency that is not present on eventlog1001, and didn't error out during build [15:14:24] dunno what's up with that! [15:14:33] !log restarting eventlogging back with pykafka 2.1.1 [15:14:57] elukey: we'll just have to manually restart eventlogging for now, if you want to restart a broker [15:15:04] actually........ [15:15:13] i might be able to hack the bugfix into the code on eventlog1001... [15:16:03] mmmmmmm [15:25:13] ok, elukey, i just hack-patched the pykafka code installed on eventlog1001 with the bugfix [15:25:23] its just a matter of a misplaced try/except outside of a loop [15:25:39] I'm watching logs there for when you are ready to reboot a broker [15:28:10] ottomata: going to step away from keyboard for ~30mins, I'll restart kafka1014 in a bit [15:28:17] k [15:30:57] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [15:31:12] Analytics-EventLogging, Analytics-Kanban: Upgrade pykafka to v2.2 - https://phabricator.wikimedia.org/T126075#2035278 (Ottomata) I just built pykafka 2.2.0, installed on eventlog1001, and rebooted eventlogging. It didn't work! There is some gevent dependency I need to figure out. I have rolled back to... [15:38:14] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [15:49:03] ottomata: in case pykafka upgrade fails before next broker reboot: I had to restart EL both when broker went down, AND up [15:53:19] hmm, interesting [15:53:21] ok good to know [15:53:46] ottomata: very weird, but that's hopw it went :( [15:54:23] ottomata: you can see 2 spikes on EL charts around 13:30 UTC [16:03:10] ottomata, do you have 3 mins for oozie error? [16:03:20] sorry ottomata, I meant joal [16:09:16] sorry mforns didn't noticed [16:09:19] I have time :) [16:09:22] cave? [16:09:30] ok joal ! [16:09:44] omw [16:15:42] back! [16:22:47] * elukey likes the new format of the Burrow's emails \o/ [16:30:58] madhuvishy: standuppp [16:55:09] Analytics-Kanban: Make last access data public - https://phabricator.wikimedia.org/T126767#2035602 (Nuria) [16:56:28] Analytics-Kanban: Make visualization of last access data - https://phabricator.wikimedia.org/T126768#2035611 (Nuria) [17:00:23] Analytics, Analytics-Kanban, Pageviews-API: Pageviews's title parameter is impossible to generate from wiki markup - https://phabricator.wikimedia.org/T127034#2035628 (Milimetric) a:Milimetric [17:00:44] Analytics, Analytics-Kanban, Pageviews-API: Pageviews's title parameter is impossible to generate from wiki markup {melc} - https://phabricator.wikimedia.org/T127034#2035641 (Milimetric) [17:00:48] Analytics-Kanban: Pageview API not dealing with url quoting very well {melc} - https://phabricator.wikimedia.org/T126669#2035642 (Milimetric) [17:11:05] (PS2) Mforns: Improve the format of the browser report [analytics/refinery] - https://gerrit.wikimedia.org/r/271253 (https://phabricator.wikimedia.org/T126282) [17:11:29] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery] - https://gerrit.wikimedia.org/r/271253 (https://phabricator.wikimedia.org/T126282) (owner: Mforns) [17:19:54] mforns: i will revert https://gerrit.wikimedia.org/r/#/c/270789/ [17:20:01] so there is no confusion [17:20:08] nuria, ok, I'm debugging with joseph [17:20:08] (PS1) Nuria: Revert "Move location of pageview and projectview archives" [analytics/refinery] - https://gerrit.wikimedia.org/r/271293 [17:24:46] Analytics-Kanban, Patch-For-Review: Move archive/pageview and archive/projectview into archive/pageview_aggregates {hawk} - https://phabricator.wikimedia.org/T127000#2035722 (Nuria) Open>declined [17:31:29] Analytics, Analytics-EventLogging: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2035752 (ori) NEW [17:31:37] elukey: lemme know when you are about to reboot a broker so I can watch EL logs [17:32:19] ottomata: still waiting for somebody to check my deployment for mw :) [17:32:29] aye k :) [17:33:07] Analytics: Add pivot parameter to tabular layout graphs {lama} [? pts] - https://phabricator.wikimedia.org/T126279#2035759 (Milimetric) [17:34:04] Analytics, Operations, Traffic: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2035781 (Milimetric) [17:34:11] Analytics, Analytics-EventLogging: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2035783 (ori) p:Triage>High [17:34:16] Analytics-Cluster, Analytics-Kanban: Upgrade to CDH 5.5 - https://phabricator.wikimedia.org/T119646#2035787 (Ottomata) [17:34:18] Analytics: Make beeline easier to use as a Hive client {hawk} - https://phabricator.wikimedia.org/T116123#2035786 (Ottomata) [17:34:29] Analytics: Make beeline easier to use as a Hive client {hawk} - https://phabricator.wikimedia.org/T116123#1740922 (Ottomata) Let's wait til we upgrade to CDH 5.5 to do this. [17:35:23] Analytics, Operations, Traffic: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2035791 (Milimetric) p:Triage>High [17:41:47] Analytics-Cluster, Analytics-Kanban: Upgrade to CDH 5.5 {hawk} [8 pts] - https://phabricator.wikimedia.org/T119646#2035810 (Milimetric) [17:42:33] Analytics: Make visualization of last access data - https://phabricator.wikimedia.org/T126768#2035811 (Milimetric) p:Triage>Normal [17:43:04] Analytics: Make visualization of last access data {mole} - https://phabricator.wikimedia.org/T126768#2035813 (Milimetric) [17:50:43] Analytics-Kanban: Make last access data public {mole} [13 pts] - https://phabricator.wikimedia.org/T126767#2035853 (Milimetric) [17:55:01] Analytics-Kanban: All members of analytics team need to have sudo -u hdfs on cluster {hawk} [2 pts] - https://phabricator.wikimedia.org/T126752#2035860 (Milimetric) a:Nuria [17:58:26] Analytics, Analytics-Kanban, Pageviews-API: Caching on pageview API should be for 1 day - https://phabricator.wikimedia.org/T127214#2035871 (Nuria) NEW [18:01:19] milimetric, u ok? [18:02:04] Analytics, Analytics-Kanban, Pageviews-API: Caching on pageview API should be for 1 day - https://phabricator.wikimedia.org/T127214#2035893 (Nuria) Caching is done on REST base , see if it is possible to be changed only for pageview API [18:04:10] Analytics, Analytics-Kanban, Pageviews-API: Caching on pageview API should be for 1 day [1pts] - https://phabricator.wikimedia.org/T127214#2035899 (Nuria) [18:05:03] Analytics-EventLogging, Analytics-Kanban: Upgrade pykafka to v2.2 - https://phabricator.wikimedia.org/T126075#2035907 (Ottomata) I just built a python-gevent 1.1b6 package, and installed it and pykafka 2.2.0 on deployment-eventlogging03 and restarted eventlogging. Let's watch it there for a few days befo... [18:05:06] Analytics-Kanban, Operations, Ops-Access-Requests: All members of analytics team need to have sudo -u hdfs on cluster {hawk} [2 pts] - https://phabricator.wikimedia.org/T126752#2035908 (Dzahn) [18:08:23] Analytics, Analytics-Kanban, Pageviews-API: Caching on pageview API should be for 1 day [1pts] - https://phabricator.wikimedia.org/T127214#2035915 (Nuria) [18:08:25] Analytics-Kanban: Pageview API not dealing with url quoting very well {melc} - https://phabricator.wikimedia.org/T126669#2035914 (Nuria) [18:13:58] ottomata: Could you restart hue (seems stuck ...) [18:16:51] Analytics-Kanban: Pageview API not dealing with url quoting very well {melc} - https://phabricator.wikimedia.org/T126669#2035945 (Nuria) Adding fix + battery of unit tests [18:18:14] Analytics-Kanban: Pageview API not dealing with url quoting very well {melc} [8 pts] - https://phabricator.wikimedia.org/T126669#2035947 (Nuria) [18:18:31] Analytics, Analytics-Kanban, Pageviews-API: Caching on pageview API should be for 1 day [1 pts] - https://phabricator.wikimedia.org/T127214#2035948 (Milimetric) [18:22:58] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2035957 (Nuria) [18:39:02] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2035999 (Nuria) - upgrade schema to include IP - do not use IP unless schema list it - client has a whitelis... [18:41:51] Analytics, Dumps-Generation: Provide a way to check if a dump has been generated - https://phabricator.wikimedia.org/T126808#2036013 (Milimetric) p:Triage>High [18:42:04] Analytics, Dumps-Generation: Avoid duplication of effort when processing dumps somehow - https://phabricator.wikimedia.org/T126809#2036016 (Milimetric) p:Triage>High [18:43:42] a-team: chrome keeps pushing me out of the hangout, I am going to drop :( [18:47:20] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} {21 pts] - https://phabricator.wikimedia.org/T126366#2036024 (Nuria) [18:47:31] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} {21 pts] - https://phabricator.wikimedia.org/T126366#2012321 (Nuria) a:madhuvishy [18:47:47] Analytics, Analytics-Kanban, Pageviews-API: Pageviews's title parameter is impossible to generate from wiki markup {melc} - https://phabricator.wikimedia.org/T127034#2036035 (Umherirrender) I was looking here: https://tools.wmflabs.org/pageviews#project=ru.wikipedia.org&pages={{FULLPAGENAMEE:Википеди... [18:51:40] ottomata: oozie is stuck (not only hue :( [18:52:24] joal: sup with the last access monthly job [18:52:27] related? [18:52:37] ottomata: Duno [18:52:43] or did you go to check what happened and found it unresponsive :) [18:52:44] madhuvishy: sorry, duno :) [18:53:19] Was lloking for mforns jobs - unresponsive, then went looking for monthly job, unresponsive as well [18:53:35] ahh [18:57:08] hmm, joal ja? [18:57:18] heya [18:57:23] oozie stuck :( [18:57:27] an27 is not actually that busy [18:57:27] hmmm [18:57:43] i just got a response from oozie jobs -jobtype bundle [18:58:10] oozie job -info 0001822-160202151345641-oozie-oozi-B [18:58:12] seems fine... [18:58:13] ? [18:58:14] talk with you tomorrow!! [18:58:15] o/ [18:58:17] laters! [18:58:42] finally got an answer to mine as well (after minutes) [18:58:47] mwargf [18:58:52] bye elukey ! [18:58:59] Sorry for disturbance ottomata [19:00:59] Analytics-EventLogging, Analytics-Kanban: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2036107 (Nuria) [19:01:07] np! [19:01:17] ok, running out for late lunch and swellbow examination [19:01:23] bye ! [19:01:33] "No data found for the page" https://tools.wmflabs.org/pageviews/#start=2015-10-01&end=2016-01-31&project=ua.wikimedia.org&platform=all-access&agent=user&pages=Вікіманія_2015 [19:01:35] End of day for me ! [19:01:43] ori: where was that mess of a discussion about the pageview api and stats.grok.se? [19:01:44] Are chapter wikis excluded from the raw data? [19:01:55] I wanted to follow up and I completely failed to find it [19:02:03] Nemo_bis: Yes, on purpose [19:02:22] ok... [19:02:29] (I vaguely recall we already discussed this) [19:02:41] milimetric, i don't think it will be a problem come to think of it [19:02:42] Nemo_bis: Ironholds arguments mostly :) [19:02:52] simply because it is not being placed on Wiki pages, but on Talk pages [19:03:01] a-team, See you tomorrow ! [19:03:13] milimetric, so the load will still be bearable [19:03:16] I have re-launched the monthly job and will continue to double check [19:03:17] yurik: but the graph gets created when the edit is made, by graphoid, no? [19:03:26] not when it's viewed? [19:03:47] milimetric, what do you mean by graph being created? [19:03:58] graph (in this case) does not contain the data [19:04:08] graphoid pulls in the data (in this case by hitting the API), and renders the image [19:04:14] correct [19:04:20] and than that image is cached [19:04:23] by varnish [19:04:31] then [19:04:35] right, but that happens when the edit is made, not when the page is first viewed since the edit was made [19:04:53] nope, it happens on request [19:05:00] oh, then we have no problem [19:05:03] yep :) [19:05:08] k, good :) [19:05:33] I'm working on the title bug now (it's not as simple as you think and it's also blocked by the conversation with Marko) [19:05:37] but we'll get it done soon-ish [19:05:55] then you can announce your graph and become world famous or whatever it is you're trying to accomplish :P [19:10:13] milimetric, i'm already world famous - i posted about it on the wiki :) Thing is, without that bug fix, i can use the incorrect encoding which does not work with any pages that have slashes [19:10:17] still better than nothing :) [19:10:35] first page that used it - female genital mutilations [19:10:36] figures [19:11:52] (CR) Nuria: Improve the format of the browser report (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/271253 (https://phabricator.wikimedia.org/T126282) (owner: Mforns) [19:45:56] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} {21 pts] - https://phabricator.wikimedia.org/T126366#2036323 (Tbayer) [19:46:10] Analytics-EventLogging, Analytics-Kanban: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} {21 pts] - https://phabricator.wikimedia.org/T126366#2012321 (Tbayer) (Cf. earlier discussion at https://phabricator.wikimedia.org/T119144#1820035 and subs... [19:58:09] (Abandoned) Nuria: Change validation message after uploading a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/261793 (https://phabricator.wikimedia.org/T76914) (owner: PranavK) [20:20:57] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Upgrade pykafka to v2.2 [5 pts] - https://phabricator.wikimedia.org/T126075#2036424 (Ottomata) [20:21:42] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Upgrade pykafka to v2.2 [5 pts] - https://phabricator.wikimedia.org/T126075#2004052 (Ottomata) > I have monkey patched the pykafka code in installed pykafka on eventlog1001 in /usr/lib/python2.7/dist-packages/pykafka/cluster.py BTW, in case s... [22:16:46] madhuvishy: can you think of a reason why the last access monthly job of january wouldn't have run? [22:17:18] madhuvishy: cause properties set it to run from dec: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/last_access_uniques/monthly/coordinator.properties [22:21:11] ottomata: yt? [22:21:32] yes [22:22:19] nuria: wassup? [22:22:37] ottomata: how can i know why the last access monthly job hasn't run? hue is down [22:22:46] and me proxying to 1001 is not working [22:23:04] ottomata: using ssh -N bast1001.wikimedia.org -L 8088:analytics1001.eqiad.wmnet:8088 [22:24:20] nuria: try stat1002.eqiad.wmnet instea of bast1001.wikimedia.org [22:24:39] but, i'd look in hue... [22:25:03] ottomata: ohhhhh [22:25:10] ottomata: now works, will look at jobs thnaks [22:25:29] (if it works...) [22:25:30] :/ [22:28:36] oozie is being slOoooow [22:53:30] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Upgrade analytics and beta project Analytics Clusters to CDH 5.5 [8 pts] - https://phabricator.wikimedia.org/T127115#2036885 (Ottomata) analytics-labs-hadoop cluster upgraded! In the process I made https://etherpad.wikimedia.org/p/analytics-cdh5... [23:07:25] nuria: sorry let me look [23:10:51] nuria: It claims to be running now [23:11:20] may be joal restarted it before