[02:39:01] Analytics-Kanban, Patch-For-Review: Caching on pageview API should be for 1 day - https://phabricator.wikimedia.org/T127214#2053972 (Pchelolo) Created a PR for frontend #restbase (https://github.com/wikimedia/restbase/pull/523) to pass through any headers sent by AQS without modifications. [07:22:42] leila, yt? [07:29:18] Ironholds: yup [07:42:50] Ironholds: I think I'm going to catch some sleep. See you later. [08:15:37] Analytics, Pageviews-API: Enable pageviews on nl/be.wikimedia.org - https://phabricator.wikimedia.org/T127804#2054400 (Romaine) [08:57:32] Analytics-Tech-community-metrics, Phabricator: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#2054463 (Aklapper) p:Low>Lowest [09:02:19] Analytics, Pageviews-API: Enable pageviews on nl/be.wikimedia.org - https://phabricator.wikimedia.org/T127804#2054400 (Sjoerddebruin) AFAIK all projects are included in the pageviews API. This seems to be something with the tool instead, so https://github.com/MusikAnimal/pageviews/issues would be a bette... [09:21:49] Analytics, Pageviews-API: Enable pageviews on nl/be.wikimedia.org - https://phabricator.wikimedia.org/T127804#2054400 (JAllemandou) Hi, Not all projects are included in the pageview API. The definition of what is called a "pageview" can be found [[ https://meta.wikimedia.org/wiki/Research:Page_view | her... [09:26:37] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2054534 (jcrespo) 205 million rows purged. [09:47:48] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054571 (Nikerabbit) [09:49:40] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054590 (Arrbee) This request is approved. Thanks. [11:53:19] Analytics-Kanban: Make last access data public {mole} - https://phabricator.wikimedia.org/T126767#2022824 (JAllemandou) a:JAllemandou [12:59:25] hi a-team :] [12:59:32] morning mforns :) [12:59:40] hey joal! [13:00:16] Actually mforns, good afternoon :) [13:00:58] hehehe joal, yes it seems morning to me too :] [13:01:56] mforns: thanks for your review on pageview sanitization ! [13:02:12] joal, np [13:02:16] mforns: other topic: stat1002 is very overloaded :) [13:02:16] :] [13:02:20] aha [13:02:33] should we do something? [13:02:35] mforns: just so that you know [13:02:52] mforns: mostly due to ezachte scripts, and I don't want to kill tham [13:02:53] ok joal I'm not doing anything on it right now [13:18:52] (PS1) Mforns: Add all reportupdater files to this dedicated repo [analytics/reportupdater] - https://gerrit.wikimedia.org/r/272712 (https://phabricator.wikimedia.org/T127327) [13:26:18] mforns: question for you [13:26:22] yes? [13:26:35] mforns: your oozie code on browser/os has been merged, right ? [13:26:40] yes [13:26:50] mforns: we need a deploy and start, right ? [13:27:03] yes, I wanted to ask you about backfilling [13:27:26] and what to do with the old files [13:27:39] hmhm ... [13:27:53] Backfilling sounds ok (if we accept that it'll take time) [13:28:04] about the old files, I am not opinionated [13:28:30] I don't think those old files have much users [13:28:48] and nuria says it would be confusing to keep them both [13:28:59] so, I think we should remove them, once we have backfilled [13:29:03] mforns: then deletion is the way I guess :) [13:29:03] if we do [13:29:13] ok cool [13:29:24] do you think we can backfill until the reports start? [13:29:37] they start at... [13:30:11] 2015-08-30 [13:30:38] the thing is... the reports are weekly, but the table is daily, so more backfilling to do [13:31:04] it's ~6 months [13:31:42] iirc the job runs in a couple minutes for one day, like 6 minutes [13:31:51] mforns: I think we can go for daily backfilling from 2015-06 (which is the date where user_agent_map gets added to the table) [13:32:32] mforns: As long as we don't put too much parallelisation, backfilling will just be an ongoing thing until it has caught up [13:32:58] so, 9 months, 270 days * 6 minutes = 1620 minutes = 27 hours of computation [13:33:26] aha makes sense, doesn't seem too long [13:34:13] joal, so do you want me to deploy? [13:34:29] mforns: sure ! [13:34:35] mforns: do you have hdfs rights ? [13:34:39] not sure [13:34:46] there's a request [13:34:53] yeah, I remember that [13:34:55] but I think it's not done yet [13:35:09] ok, so you do the tin deploy, I do the hdfs one :) [13:35:09] * mforns looks [13:36:06] yes! https://phabricator.wikimedia.org/T126752 [13:36:25] awesome :) [13:36:27] I can do both, but I have to look at the docs first [13:36:30] I'll watch you do then :) [13:36:34] batcave? [13:38:48] sure mforns, joining [13:50:06] ottomata: Hi sir, are you here ? [13:53:35] mforns: https://gist.github.com/jobar/9126574693ece814a2ac [14:04:24] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2055302 (ema) p:Triage>Normal [14:05:14] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054571 (ema) [14:09:17] joal: good morning! [14:09:22] eating some breakfast then ready to get started [14:09:23] hey ottomata [14:09:39] in meeting with halfak, so no rush is good ;) [14:09:46] k [14:09:55] also ottomata : stat1002 is overloaded :( [14:10:03] Don't know if you can do anything [14:13:45] looking [14:14:47] looks like ezachte is doing some crunchin :) [14:15:03] joal: is it causing a problem? [14:15:14] ottomata: can't deploy, and almost can't use hive [14:15:25] ottomata: would be good to nice those things :) [14:15:39] But ezahte not on IRC [14:16:13] hm [14:16:14] yeah [14:16:35] will email and ask him to nice [14:18:08] ok, I could have done that byt didn't really know what channel to use :( [14:19:29] i'm just personal emailing him, CCing you [14:19:38] thx ottomata :) [14:34:42] joal: getting ready, i'm going to just suspend all oozie jobs I can find [14:34:50] ok ottomata [14:34:57] let me know if I can help [14:35:36] Actually ottomata, I think you only need to pause the laod ones --> everything will naturaly wait after those [14:35:49] also ottomata, you should stop camus :) [14:35:58] yes [14:36:07] there are also now discovery ones! [14:36:14] True ! [14:36:19] forgot about those [14:36:27] yeah, all ours are webrequest based [14:36:35] so pausing the bundles (whihc I just did) [14:36:37] should be enough [14:36:41] i don't have to search for coordinators [14:36:47] ottomata: correct [14:37:02] Then we should wait for the dependent jobs to have finihed [14:37:08] ok, i paused all bundles, and 3 analytics-search coordinators [14:37:10] ja cool [14:37:39] i'm not going to stop camus until we are ready to shut things down [14:37:47] which will probably be in 30 mins or so [14:37:55] ok ottomata [14:43:20] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054571 (ema) Hi, it looks like @Nikerabbit is already a member of the statistics-users group. Perhaps something is wrong with the S... [14:48:36] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054571 (Krenair) statistics-users provides stat1003 access, not stat1002. It also does not provide access to the password needed to a... [14:58:36] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T127808#2054571 (Ottomata) https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Access_Groups [15:08:05] ottomata: everything smooth ? [15:08:06] joal: ja, am just waiting for last thigns to finish [15:08:06] i killed discovery jobs [15:08:06] commenting out camus now [15:08:06] ok [15:09:31] joal: as soon as runnig jobs finish, I will start with process outlined here [15:09:31] https://etherpad.wikimedia.org/p/analytics-cdh5.5 [15:09:38] will let you know each step as I go [15:10:11] ok ottomata, let me know if there is any specifics you want me to monitor [15:10:21] k [15:13:26] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to researches - https://phabricator.wikimedia.org/T127808#2055392 (Nikerabbit) [15:14:15] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests: access for nikerabbit to researches - https://phabricator.wikimedia.org/T127808#2054571 (Nikerabbit) Ottomata summarised this well: > Nikerabbit you want to be in the researchers group, and you will use it to access a file o... [15:20:07] joal: cool, finally! no jobs [15:20:09] proceeding [15:20:24] !log shutting down analytics (hadoop) cluster for CDH 5.5 upgrade [15:21:08] who [15:21:12] oops:) [15:24:25] ok, stopping analytisc1027 services [15:26:33] oops, I was just trying to run a query :) [15:29:46] stopping all hadoop services [15:41:59] just upgraded cdh on journalnode hosts, moving to master and standby [15:43:46] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2055478 (Krinkle) Fair enough, but I'm making the case we don't need statistics. This worked and was used and linked to. There's... [15:47:41] upgrading workers [15:48:01] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2010000 (Krenair) I don't care about this redirect enough to upload the patch, but I imagine this is because stats.wikipedia.org... [15:57:14] hadoop back up, testing mapreduce job [16:03:20] upgrading packages on analytics1027 [16:08:36] upgrading oozie db and sharelib [16:11:40] starting analytics1027 mysql migration to analytics1015 [16:23:38] installing hive and oozie servers on analytics1015 [16:25:14] ottomata: o [16:25:16] o/ [16:25:25] hiyaa [16:25:29] how is the migration going?? [16:25:50] perfectly so far! [16:26:05] i'm doing the most risky (but still not very) part now, moving hive and oozie to analytics1015 [16:26:10] i already did the db move [16:26:15] promoted slave to master, etc. [16:26:32] will reboot analytics1027 soon [16:28:18] super! [16:30:24] ah, just found something out, spark-core now depends on flume-ng, which is not in our apt... [16:30:26] gotta add it [16:30:46] a-team: standddupppp [16:31:18] eee! [17:00:50] mforns: you got a sec? [17:00:56] milimetric, sure [17:01:02] to the batcave! [17:01:05] o/ [17:02:11] !log rebooting analytics1027 for kernel upgrade [17:02:27] Hey ottomata, which statsd lib would you recommend me using for python ? [17:02:50] joal: i don't really have an opinion, i guess whatever we have in apt [17:02:56] or is in debian [17:03:03] hm [17:03:31] looks like python-statsd 3.0.1 in trusty at least [17:03:39] right [17:03:44] just 'statsd' in python [17:03:45] i thikn [17:04:01] python-statsd seem to be widely use, let's go for that :) [17:04:04] Thqanks [17:04:39] k [17:05:49] hm, thinkg about that again: statsd is for time-regular info at second level - Here we plan to have daily data [17:05:52] ottomata: --^ [17:06:00] joal: sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.sendto(b'foo.bar:123|ms', ('statsd.eqiad.wmnet', 8125)) [17:06:23] just saved yourself a library dependency [17:06:33] Thanks ori ! [17:08:33] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2055866 (Neil_P._Quinn_WMF) Thank you, @jcrespo! One questi... [17:09:47] ottomata, ori: Given that we want to plot daily data, I can [17:10:18] can't think of something else than sending data to graphite directly, with a timestamp (same as what we did for restbase hits) [17:10:24] any objections ? [17:11:47] joal: what is this for? [17:12:13] global hadoop metrics: how many jobs, mappers / reducers, how many users [17:12:18] ah [17:12:21] ja graphite sounds fine [17:12:32] Very few data (I think daily is ok) [17:12:37] ok cool [17:13:06] joal: everythign is kinda back up! [17:13:08] looking good [17:13:12] about to unsuspend oozie jobs [17:13:17] Looking with you ! [17:14:12] man, 1030 didn't upgrade via salt [17:14:18] ? [17:14:31] Well caught ! [17:14:55] ottomata: looks like I can't log in hue [17:15:39] joal: send to statsd, not graphite [17:15:56] if you send to graphite you have the following issues to deal with: [17:16:04] ori: how to handle not regular data ? [17:16:05] mforns: before you push that for review [17:16:10] milimetric, yes [17:16:14] if you don't send a metric in the minimum aggregation period, you have a hole in your data [17:16:16] ori: I want to send daily points [17:16:23] how come you put the text color (which works great btw) in the view model instead of the bindings? [17:16:37] I figured you could just use d.color in the bindings [17:16:45] if you send more than one metric in the smallest aggregation period, the lattermost wins -- they are not averaged or summed [17:16:50] milimetric, AHA [17:16:53] *aha [17:16:59] ori: the reason to use graphite for this stuff was that statsd doesn't let you send timestamps of when the data is being recorded for [17:17:26] * mforns looks [17:17:30] yeah, it's tricky :/ [17:17:54] ori: this comes up in our case more because we might rerun jobs later etc [17:18:07] i have resorted to writing a thing that writes directly to whisper files in the past [17:18:10] ori: we are not trying to have complete datasets in the way statsd does, we are trying to get very sparse data (daily points) and we graph them using tricks in grafana [17:18:13] oh! mforns never mind, I don't think that's possible... [17:18:19] i've had grand plans of setting up influxdb on tungsten but haven't gotten around to it; it is supposedly better at this [17:18:26] aah [17:18:28] milimetric, I don't know [17:18:37] 'cause the text is created as a sibling of the polygon [17:18:37] why no? [17:18:46] ah [17:18:50] (if you have time / interest for trying it out I can pass that machine over to you) [17:19:09] ori: sending to graphite directly seems to be mostly okay though - although you don't get the nice sum, count graphs etc [17:19:11] also we were considering opentsdb at one point too no? [17:19:33] mforns: yeah, don't worry about it for now, that looks like it needs some refactoring to all happen under d3's control, but maybe some other time :) [17:20:36] milimetric, ok, then we can refactor both hierarchy and sunburst [17:20:38] ori: I have no idea what opentsdb is - sure I'd love to try playing with influxdb - may be next week or so [17:20:54] yes, they both do the same thing [17:21:05] ori: IIRC, invetigation has been made by addshore [17:21:18] ok [17:21:32] opentsdb was interesting to look at, but we only glanced [17:22:06] ori: , joal, we have talked about a project to use some fancy time series db / graph tool for analytics data, maybe in analytics cluster [17:22:08] but have put it off [17:22:23] as it is not a high prioirty, and is mostly wanted for internal project usage graphs [17:22:27] would be cool though! [17:22:28] biggest diff between influx and openTSDB (to my opinion) is that the latter runs on the cluster (good for big data, bad for possible downtimes) [17:22:42] for sure ottomata :) [17:23:02] great! no more cdh 5.4 packages anywhere i can see [17:23:05] ottomata: oozie is responsive (while doing nothing, it's good ;) [17:23:07] godog is also a big fan of prometheus [17:23:12] cool [17:23:14] hive looks good too i think [17:23:16] iirc all of these are configurable as backends for grafana [17:23:24] yup ottomata, checked it and it looked ok [17:23:32] joal: ok with you if I unsuspend oozie jobs? [17:23:43] ottomata: hue thing first ? [17:23:48] Can't log in :( [17:23:50] hue thign? [17:23:53] no? i logged in [17:23:58] shell username + ldap pw? [17:24:11] AH! [17:24:18] hue users all unsynced?!?! [17:24:26] bah! [17:24:26] ha [17:24:38] brb [17:25:23] ottomata: camus already running ;) [17:28:36] *reads up* [17:30:18] ja i wanted to run puppet so i uncommented crons [17:30:19] milimetric, there is another thing I'd like to change... [17:30:35] when you zoom across a couple levels, the chart shrinks... [17:30:46] ottomata: still no chance with hue :( [17:30:47] it should stay the same size [17:30:49] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2055964 (jcrespo) I will manually make sure to resync it- a... [17:31:34] joal: ja han gon [17:31:38] checking a few things [17:31:43] ottomata: sure [17:32:00] * milimetric looks for this shrinky chart thing [17:32:31] mforns: oh yeah, I noticed that but I liked it [17:32:44] it's cool, gives you an idea that something's happening in case you maybe missed the zoom [17:32:57] milimetric, I see... [17:33:05] like your cat jumps in your lap and ON PURPOSE turns off your monitor [17:33:06] ok :] [17:33:24] xD [17:33:36] * milimetric ::grumbles about the cat:: [17:34:31] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2055990 (jcrespo) Regarding replication lag, I shared a scr... [17:36:17] joal: i'm going to spend a few minutes seeing if i can quickly make the hue ldap integration cooler [17:36:23] can you help me with something? [17:36:38] can you prepare a puppet patch that increases heap sizes of hive and oozie servers? [17:37:09] should be in hiera [17:37:10] files like [17:37:27] hieradata/eqiad/cdh/{hive,oozie}/....yaml [17:37:29] heapsize: [17:37:31] etc. [17:37:38] i can do, but i'm looking at hue real quick [17:37:41] no worries if not [17:38:45] (PS11) Mforns: Add hierarchy visualization [analytics/dashiki] - https://gerrit.wikimedia.org/r/269713 (https://phabricator.wikimedia.org/T124296) [17:38:51] joal: Let's ask analytics e-mail list (public) feedback on how to release the unique devices dataset [17:38:51] milimetric, ^ [17:39:19] joal: that way we have input as to what people would like to see, will try to craft an e-mail [17:39:24] (CR) Milimetric: [C: 2 V: 2] "nice and clean, love it" [analytics/dashiki] - https://gerrit.wikimedia.org/r/269713 (https://phabricator.wikimedia.org/T124296) (owner: Mforns) [17:39:34] :] [17:41:56] ottomata: I can ! [17:42:00] Was away for a minute [17:43:00] ok nuria [17:48:03] ottomata: hive and oozie puppet code is in submodule ? [17:49:46] found it ottomata [17:51:20] yes but you only need to set it in hiera [17:51:33] (IF it works...) [17:51:41] joal: , try hue now. [17:51:42] oof [17:51:50] i had to manually resync each user in analytics-privatedata-users :/ [17:51:59] gonna make a task to fix this ! [17:52:00] ottomata: Worked ! [17:52:07] Thanks mate [17:52:32] ottomata: about HEAP_SIZE, I don't get how the thing get's passed to a template or something :( [17:52:48] hieradata defined values that are reused in puppet conf, right? [17:54:10] Analytics, Analytics-Cluster: Improve Hue user management - https://phabricator.wikimedia.org/T127850#2056054 (Ottomata) [17:54:11] hehehe [17:54:21] yeah hiera is wayyyy high up [17:54:35] ok, joal i'm through with hiera, i can do it, unless you really want to :) [17:54:40] sorry [17:54:42] through with hue* [17:54:51] ottomata: I'd like to learn :) [17:54:57] ok [17:55:07] hiera is all about scope [17:55:09] so uhhh [17:55:12] in this case in production [17:55:16] cdh module parameters [17:55:22] are being filled in from hiera in the 'eqiad' scope [17:55:23] so [17:55:28] makes sense [17:55:35] hieradata/eqiad/cdh/** [17:55:43] in files named after the classes [17:55:53] with hiera keys after the parameters [17:56:12] you can see i'm already setting heapsize via hiera for hive server and metastore [17:56:27] because heapsize is a parameter on the cdh::hive::{metastore,server} classes [17:56:43] it is also a paramater on the cdh::oozie::server class [17:56:49] so, you should be able to add to the server.yaml file too [17:56:55] sorry [17:56:58] oh [17:57:05] there is no oozie server file yet [17:57:07] gotta make one [17:57:15] cdh/oozie/server.yaml [17:57:16] and add [17:57:17] heapsize: [17:57:21] not sure what we should set them at hmmm [17:57:29] about hive first: what put me in is the cdh::hive file --> didn't find it [17:57:51] cdh/hive/server.yaml [17:57:54] cdh/hive/metastore.yaml [17:57:56] cdh folder is empty for me ! [17:58:01] in hieradata? [17:58:06] hierdata/eqiad [17:58:11] hieradata/eqiad/cdh [17:58:24] no, in modules [17:58:29] naw, not in modules [17:58:34] cdh is a ubmodule [17:58:37] git submodule update --init [17:59:02] Riiiight, that was the thing I was after ! [17:59:25] Thanks, now it all should fall in place: find the parameter name, put it in hiera file with correct value :) [18:00:11] yup [18:00:36] a-team: staff on batcave right? [18:00:40] nup [18:01:34] a-team: let's do batcave [18:01:34] a-team: let's all go to batcave [18:01:50] milimetric: maybe you can cahnge meeting video link? [18:01:52] *change [18:02:06] joal, milimetric , ottomata : holaaaa [18:02:16] ottomata: heapsize parameter is defined in hive/server.pp - Shouldn't we use a specific file ? [18:02:44] ? [18:02:57] heapsize is set via hiera in hieradata/eqiad/cdh/hive/server.yaml [18:03:08] joal: ^ [18:03:13] yessir, makes sense [18:03:25] got it now ! [18:03:33] pffff, was looking at hive.yaml [18:03:36] :( [18:03:40] sorry [18:03:44] * joal is learning [18:04:13] ottomata: shall we bump that to 4096 for hive and same for oozie ottomata ? [18:04:25] looking at recomendations [18:04:31] http://www.cloudera.com/documentation/enterprise/5-4-x/topics/admin_hive_configure.html [18:05:25] lets go with 6 and 10G for hive server and metastore [18:05:33] ottomata: let's go for large, and make it 6144? [18:06:39] joal: oozie server 2G? 4G? [18:07:03] hm ... Checking current value [18:08:06] Analytics-EventLogging, Analytics-Kanban: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2056154 (Nuria) We need to verify that udp traffic can get from dallas to eqiad OR migrate eventlogging server to use kafka (as equiad will be up while this exercise is t... [18:09:05] ottomata: currently, catalina mem is set to 1024 in oozie-env [18:09:24] ja [18:10:01] Not really more users, just possibly better repsonse times --> 2048 ? [18:11:01] Analytics-EventLogging, Analytics-Kanban: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2056158 (Nuria) Can server side produce via http to varnishkafka [18:11:24] k [18:22:21] (CR) Bearloga: "Sure, I'll get those performance numbers updated. As for insource thing, there's two because one checks if the query is insource: in gener" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/254461 (https://phabricator.wikimedia.org/T118218) (owner: Bearloga) [18:34:48] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2056335 (Neil_P._Quinn_WMF) @jcrespo, ah, okay, I misunders... [18:37:23] joal: cool, it worked! [18:37:43] great ottomata [18:38:28] you should have access now too [18:38:38] uhh in maybe 1 min [18:38:44] ok joal awsome [18:38:52] ready to unpause oozie stuff? [18:39:09] ottomata: sure, LET'S DO IT ! [18:39:37] ottomata: you do it, or shall I ? [18:39:41] i will [18:39:51] ok waiting for the flow to reappear in hue :) [18:40:22] man, ellery must be happy, he has a full cluster for himself currently :) [18:40:32] haha [18:40:51] hmmm [18:41:14] hang on, an27 is funky, hmmm [18:41:23] with oozie/hive settings, puppet shoulda fixed, hmm [18:42:33] ottomata: no hive/oozie processes on an1027, only jmxtrans [18:42:38] i just stopped them [18:42:51] ottomata: and by the way, no jmxtrans on an1015 [18:42:57] ok ottomata makes sense :) [18:43:06] oh [18:43:07] hm [18:43:14] hmmm, yeah i think we don't have it configured for hive or oozie [18:43:20] ok cool [18:43:28]