[01:44:53] 10Analytics, 06Performance-Team: Check if the EventLogging User Agent schema upgrade breaks any performance tool/metric - https://phabricator.wikimedia.org/T156760#3004111 (10Nuria) >Why keep it as a string, though? It should be easy enough to bump the EventCapsule schema and declare this as an object instead.... [01:59:24] 10Analytics: Investigate rise in IE views from Pakistan since 2015 - https://phabricator.wikimedia.org/T157404#3004144 (10Tbayer) [02:04:39] 10Analytics, 06Performance-Team: Check if the EventLogging User Agent schema upgrade breaks any performance tool/metric - https://phabricator.wikimedia.org/T156760#3004148 (10Krinkle) >>! In T156760#3004111, @Nuria wrote: >>Why keep it as a string, though? > It is a breaking change, not disputing that. Keeping... [02:06:30] 10Analytics, 06Performance-Team: Check if the EventLogging User Agent schema upgrade breaks any performance tool/metric - https://phabricator.wikimedia.org/T156760#2985302 (10Tbayer) Agree that it might be preferable to declare it as object rather than as string, and to rename the field. But shouldn't this dis... [05:31:04] 10Analytics: Import 2001 wikipedia data - https://phabricator.wikimedia.org/T155014#3004393 (10Graham87) I've already checked the so-called January 2002 dump ... It's just a UseModWiki version of the Nostalgia Wikipedia dump. [05:35:41] 10Analytics: Import 2001 wikipedia data - https://phabricator.wikimedia.org/T155014#3004409 (10Graham87) Also, re the actual 2001 dump, I think it'd be best to only import edits when there is no or only a trivial gap in the page history between the last edit in those dumps and the first surviving one in the Wiki... [07:28:12] joal: o/ [07:28:22] just restarted the NM on an1028 [07:28:31] there were 3 application masters running on it :P [07:28:57] and I don't see any killed job now [07:33:16] * elukey dances [07:34:19] the state consumes around ~5MB of disk, so it seems super good [07:34:42] the main problem is that we'd need to restart all the node managers one last time to apply the change to all of them :D [07:36:06] maybe we can temporarily stop the bundles, wait for the queues to drain, rapidly restart the NMs and then restart [07:36:15] shouldn't take more than 30 mins [09:28:23] elukey: That's awesome :) [09:28:39] elukey: Thanks a lot for having found that setting and do the changes ! [09:29:06] \o/ [09:29:11] already applied to all the NMs [09:29:22] elukey: you rock :) [09:29:24] I mean, I need to restart them [09:29:34] sure, let's start if you want :) [09:50:12] joal: need to step afk for an hour, will be back and do the work :) [09:50:17] * elukey afk for a bit! [09:50:23] Bye elukey [09:53:34] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3004775 (10JAllemandou) @chasemp / @yuvipanda : Another question for you guys - There seems to be no rev_text_id in the DBs (0 everywhere). Can you explain me why so, and if this is... [10:15:43] (03CR) 10Joal: [C: 04-1] "This version is not functional, missing some ops-network thing. Code needs to access WMF APIs, but so far can't do it because of network A" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335846 (owner: 10Joal) [10:27:11] (03PS16) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) [10:35:16] (03PS17) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) [11:16:28] elukey: There really something weird with archiva - my build takes infinte time to download [11:23:22] joal: mmmm but is it downloading at very low speed or is it blocked? [11:23:31] again on the same pom? [11:23:41] elukey: on every pom now [11:24:12] elukey: ~8Kb/s download speed [11:25:33] ahahahah 56k [11:25:41] back to the 90s [11:26:49] joal: on which host? [11:27:05] stat1004 [11:27:35] also elukey, I think you forgot to create the task on proxy/network for analytics vlan :) [11:28:59] I discussed the issue with Mark and Daniel since they are working on it, the plan is to review the network ACLs and propose a change.. the task is not there but the work is in progress :) [11:29:46] joal: can you give me an example of archiva link that takes ages to complete? [11:29:52] it seems super fast from stat1003 [11:29:54] *1004 [11:30:11] elukey: https://archiva.wikimedia.org/repository/releases/com/cloudera/cdh/zookeeper-root/3.4.5-cdh5.7.0-SNAPSHOT/maven-metadata.xml [11:30:28] elukey: seems that resource is not existing [11:30:45] This is weird, it has been working for long time :( [11:31:24] elukey: maybe ottomata has closed the mirroring ? [11:31:58] I am completely ignorant about archiva, it is one of the dark corners of our infrastructure that I don't know anything about :D [11:32:28] hm [11:32:30] mmmm joal cdh 5.7? don't we have 5.5? [11:33:01] elukey: I need new spark version, so using CDH5.7 (will soon use CDH 5.10 [11:34:42] and is it on archiva? I mean, have you successfully built etc.. with that in the past? [11:35:08] elukey: I've built in the past yes, and now seems not working anymore :( [11:39:04] joal: let's ask to the master ottomata, I believe that something has happened and now the things that you need are not there anymore [11:39:21] elukey: yes, I think something has happened too :) [11:45:32] elukey: it finally did it -- Why .. I have no idea [11:45:47] mmmm [11:48:14] (03PS4) 10Joal: Add spark streaming job for banner impressions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/333655 [11:48:46] * elukey quick lunch and then I promise that I will not be afk anymore.. a lot of interrupts today :) [12:14:59] back [12:31:21] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005090 (10elukey) p:05Triage>03Normal [12:31:27] joal: --^ [12:32:17] Thanks a lot elukey :) [12:33:21] there is a bigger work to do for that VLAN, I am going to add some details.. [12:33:24] sigh [12:36:27] :( [12:46:12] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005127 (10elukey) Adding install1002's IP to the whitelist should be: ``` edit set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.... [12:56:07] 10Analytics: Update DataLake History schema to only contain "objective" measures - https://phabricator.wikimedia.org/T157362#3005175 (10JAllemandou) @Halfak : Does that a corrct idea to you as well ? [13:02:58] 10Analytics: Update DataLake History schema to only contain "objective" measures - https://phabricator.wikimedia.org/T157362#3005219 (10JAllemandou) [13:04:34] elukey: have you started regular NM restarts? [13:05:54] joal: whenever you want to jump on that task give me a shout ;) [13:06:15] fdans: Let's wait for halfak confirmation first :) [13:06:24] sure sure! [13:11:55] joal: nope, we can do it anytime [13:12:13] elukey: since you'll be the one doing it, you let me know :) [13:15:26] joal: I am reviewing the network acls, archiva changed IP [13:15:28] :P [13:16:11] it was reimaged from a physical box to a ganeti VM about half a year ago [13:18:17] moritzm: I am reviewing the ACLs for the analytics VLAN in https://phabricator.wikimedia.org/T157435, a lot of stale things :( [13:25:28] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005305 (10elukey) Stale things found while reviewing: * term udplog is probably not worth to keep * term kafka is missing kafka2003's IP * term archiva should contain meitnerium's IP,... [13:26:30] argh aqs, zookeeper are a bit messed up [13:26:35] (network acls) [13:32:28] 10Analytics-Tech-community-metrics: Kibana's Mailing List data sources do not include recent activity on wikitech-l mailing list - https://phabricator.wikimedia.org/T146632#3005321 (10Lcanasdiaz) The mailing list returned some errors we did not catch by mistake. According to our developers this is fixed with the... [13:55:20] joal: do you know if we are still sending traffic to the elastich search cluster? [13:55:24] from the hadoop worker nodes [13:55:43] elukey: It makes a long time I have not discussed with discovery about that, but I think yes [13:56:01] There are discovery jobs running regularly, and IIRC there are some sending data to ES [14:02:44] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005522 (10elukey) Most urgent fixes: * Remove old AQS IPs ``` delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.123/32 delete firewall family i... [14:03:26] joal: because all the es ips are whitelisted in the ACL [14:03:34] and I can see some stale ones [14:03:41] elukey: hm [14:03:58] elukey: in that respect, possibly better to ask ebernhardson or dcausse :) [14:06:20] joal: all right.. shall we do the rolling restart of NM? Is it worth to stop oozie and wait for spark jobs to complete (if possible in a reasonable amount of time) ? [14:07:24] Discovery Transfer To http://elastic1017.eqiad.wmnet:9200 [14:07:27] ahhaha lol [14:07:29] auto-answered [14:08:05] elukey: I don't think it's necessary to stop/start oozie [14:09:06] elukey: I'm gonna take a break now, so please move forward with restarting NMs, I'll check for failed jobs when coming back (before standup [14:09:11] elukey: works for you? [14:11:17] joal: all right, I'll proceed with the restarts :) [14:11:27] awesome, thanks elukey :) [14:11:33] taking a break now ! [14:17:22] you guys are so great! every time I come online in the morning the two of you are all "actions, go , execute, do it, done" :) [14:18:49] :D [14:19:58] joal & fdans: +1 [14:20:15] for removing "productive" but having the metrics to identify it as currently defined. [14:20:43] halfak: Thanks ! [14:21:37] ottomata: o/ - today I might need a review for https://phabricator.wikimedia.org/T157435 [14:22:14] maybe another pair of eyes, some of the ACLs are a bit of a mess [14:22:36] 10Analytics: Update DataLake History schema to only contain "objective" measures - https://phabricator.wikimedia.org/T157362#3005587 (10Halfak) +1 [14:23:10] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005589 (10Ottomata) +1 to all of these. But, seeing as there has been an IPv6 with the ACLs for a while, maybe we should ask Ops about the use of continuing to support this VLAN. Not... [14:23:14] yeah [14:23:31] all the changes you propose make sense elukey, i don't have the context of what the actual switches look like [14:24:22] ottomata: thanks! I am also asking to Discovery the new es hostnames [14:26:40] hey team :] [14:27:44] o/ [14:33:00] all the node managers have been restarted [14:33:16] so from now on it should be graceful restarts only [14:35:56] yeehaw [14:36:51] !log restarted webrequest-load-wf-text-2017-2-7-13 [14:36:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:38:25] cool [14:41:21] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3005658 (10elukey) Old/New elastic search IP from Discovery: https://etherpad.wikimedia.org/p/analytics-acls [14:43:24] aqs1009-a still wip [15:14:09] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#3005780 (10Ottomata) Hi yall, just curious. What's the word? :) [15:18:35] fdans: btw, I was saying vue has vuex, which is like redux: http://vuex.vuejs.org/en/structure.html [15:18:48] rather, serves the same purpose as redux [15:19:09] oh that's cool [15:19:39] I mean that's another decision to make, seeing if we need a state manager [15:29:18] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#3005834 (10mobrovac) FYI, these is also an interest in attaching ORES WP10 deltas to new revisions, e.g. {T145829}. Perha... [15:32:32] https://testing.googleblog.com/search/label/TotT [15:38:02] (03CR) 10Fdans: Adds map visualizer to Dashiki (032 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) (owner: 10Fdans) [15:40:35] (03PS4) 10Fdans: Adds map visualizer to Dashiki [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) [15:49:44] elukey: ciao :) [15:52:51] urandom: o/ [15:53:00] 06Analytics-Kanban: Document the difference in aggregate data on wikistats and wikistats 2.0 - https://phabricator.wikimedia.org/T150963#3005920 (10Nuria) 05Open>03Resolved [15:53:25] 06Analytics-Kanban, 13Patch-For-Review: Pageview Jobs: Make workflows easier to maintain using a variable instead of repeating some complex value accross the files - https://phabricator.wikimedia.org/T156668#2983116 (10Nuria) 05Open>03Resolved [15:53:25] elukey: i just found the ticket :( [15:53:32] urandom: not sure if I did the right thing with restbase-dev1001, but it was a bit weird that raid failure [15:53:37] 06Analytics-Kanban: Debian package for ua parser latest version - https://phabricator.wikimedia.org/T156821#3005926 (10Nuria) 05Open>03Resolved [15:53:54] elukey: no, i think that makes sense; thanks for taking care of it [15:53:55] 06Analytics-Kanban, 13Patch-For-Review: Update montly 'unique computation' jobs for better resource management - https://phabricator.wikimedia.org/T156921#3005927 (10Nuria) 05Open>03Resolved [15:54:10] elukey: there is no data there yet, so it wouldn't have mattered, but still... [15:54:12] 10Analytics-General-or-Unknown, 06Analytics-Kanban, 13Patch-For-Review, 07Privacy: analytics.wikimedia.org loads resources from third parties - https://phabricator.wikimedia.org/T156347#3005928 (10Nuria) 05Open>03Resolved [15:54:18] urandom: but is it intended that the srv partition is a raid0? [15:54:24] yeah [15:54:27] ahhh okok [15:54:35] now it makes more sense [15:54:37] software raid [15:54:50] sure but I thought it should have been a raid10 [15:54:57] oh, no [15:54:57] Hey ottomata :) Thanks for the nice comment :) [15:56:59] elukey: i guess this is one of those deals where you can't get around the hardware raid controller? it's presenting each drive as an array? [15:58:13] elukey: i guess this is a failed drive? [15:58:21] urandom: ah yes yes, until Chris has swapped the disk the host is unusable.. both cassandra instances are down, but we have other two racks so it should be fine (and its dev..) [15:58:26] yep! [15:58:33] gotcha [15:58:34] one drive failed, the whole raid went down [15:58:39] boo [15:58:54] but at least it's only dev, and isn't being used yet [15:59:02] yep! [15:59:41] Thanks elukey for having restarted the oozie job :) [16:00:04] elukey: and obviously for having made our NM more robust !! [16:06:07] joal: let's try after stand up to break them! :D [16:06:20] 06Analytics-Kanban, 06Operations, 06Performance-Team, 06Reading-Admin, 10Traffic: Preliminary Design document for A/B testing - https://phabricator.wikimedia.org/T143694#3005963 (10Nuria) a:03Nuria [16:15:01] milimetric: We have new datasets we'd like to add a dashboard for - we need a config page on meta, new scripts in limn-multimedia-data, and...isn't there one more thing? [16:17:55] marktraceur: yeah, we'd have to compile and deploy the static site that would render your dashboard [16:18:16] so send me the config and I'll show you what I do with it [16:18:26] (and the dashiki layout you want to use) [16:18:53] (and what you want the new dashboard to be called, I assume edit-analysis.wmflabs.org/<>) [16:23:44] joal: I am restarting node managers at random, hitting the ones will app masters [16:28:44] elukey: awesome [16:28:51] elukey: don't kill them all though ;) [16:28:57] I restarted 9 hosts, all good for the moment [16:29:48] * elukey dances [16:30:56] joal, quick gzip? [16:31:00] mforns: Yes ! [16:31:03] mforns: batcave? [16:31:18] joal, batcave-2 [16:31:22] sure [16:31:30] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [16:42:03] (03CR) 10Milimetric: "when you take a look at this, as you read through the comments it's useful to mark them as "Done" or reply if you disagree. That way we c" (037 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) (owner: 10Fdans) [16:42:44] fdans: only one thing, the double-wrapped observables, the rest is syntax stuff you missed in my last comments [16:47:36] milimetric: thank you for that :) the observables was me being daft and not saving&committing [16:48:49] oh yeah, I've done that a lot. What I do now is: [16:48:49] 1. read comment [16:48:49] 2. fix code, commit --amend [16:48:49] 3. mark comment "Done" [16:49:45] elukey: this NM restart change is AWESOME :) Thanks a lot again :) [16:51:01] milimetric: right, so you hit "done" once you've committed the fix [16:52:15] yep, makes it so I don't have to think (/me lazy) [16:53:06] (03CR) 10Gedda: [V: 032 C: 032] Amend name for Goan Konkani [analytics/wikipagestats] - 10https://gerrit.wikimedia.org/r/335030 (https://phabricator.wikimedia.org/T126148) (owner: 10Gerrit Patch Uploader) [16:53:30] milimetric: It's matthiasmullie's thing, I'll get him in here [16:53:38] k [17:10:59] milimetric: I guess he's still busy with some other things, but he knows where to find you :) [17:11:10] scary :) [17:14:31] a-team: fyi chad is now.... ahem ...RainbowSprinkles on irc [17:14:57] My secret is out [17:15:27] RainbowSprinkles: while I have you here, I wanted to deploy Dashiki to meta now that it's approved by the community [17:15:38] I don't have deploy rights, who should I bug? [17:15:47] Make a patch and put it on swat [17:17:17] RainbowSprinkles: from that page, "Forbidden types of patches: No new features/extensions" [17:18:04] Ah, ok. Um, file a task and write a patch for it and we'll find a window :) [17:18:24] I can probably handle it [17:19:05] 06Analytics-Kanban, 10Wikimedia-Extension-setup: Deploy mediawiki-Dashiki extension to meta.wikimedia.org - https://phabricator.wikimedia.org/T156971#3006157 (10Milimetric) [17:25:36] milimetric: Ah, I suppose we'll need to run namespaceDupes.php to fix up the pages to the right place [17:26:20] RainbowSprinkles: oh it's ok, I can do that manually because I have to sort through them anyway. They're not quite dupes, they're going from Config: to Config:Dashiki: [17:26:50] (I'm currently trying to figure out how to edit InitializeSettings.php for just meta (no spoilers!) [17:26:54] ) [17:39:49] (03PS5) 10Fdans: Add map visualizer to Dashiki [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) [17:41:39] (03CR) 10Fdans: Add map visualizer to Dashiki (037 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) (owner: 10Fdans) [17:44:10] RainbowSprinkles: I don't wanna cut in line or bother you, what's the right step after this patch? https://gerrit.wikimedia.org/r/#/c/336444/ [17:44:19] * RainbowSprinkles looks [17:45:22] At a quick glance, that looks right. So assuming I do it, we'll find a window, sync the files and make sure they all look good on meta. [17:45:42] Afterwords, as cleanup, we can remove the labs-specific enabling of it if possible :) [17:46:43] RainbowSprinkles: definitely, I'll do a follow-up patch now [17:48:23] RainbowSprinkles: most things have default => true so I couldn't find an example of a beta cluster => true exception to use in wmgEnableDashikiData [17:50:04] is it "deploymentwiki"? [17:50:31] No. What you'd do on beta is still leave the InitialiseSettings-labs.php bit, but remove CommonSettings-labs.php [17:50:44] But that's unimportant for later. default => false, foowiki => true is legit [17:53:32] gotcha [17:56:07] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 4 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#3006265 (10Jdlrobson) [17:56:42] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 4 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2973889 (10Jdlrobson) Feel free to continue bumping this up after T156680 is resolved [17:56:52] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 4 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#3006272 (10Jdlrobson) [17:57:08] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 3 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2973889 (10Jdlrobson) [17:57:52] do we have staff meeting or is my cal funky? [17:59:02] we do [17:59:55] ottomata: https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/software/varnish/varnishkafka [17:59:58] :O [18:00:16] we have pull requests for varnishkafka [18:00:28] Riccardo was chatting with people in Fosdem [18:00:45] we didn't realize about them since we are not put in the code reviews automagically [18:01:10] oh wow [18:03:04] those patches look good! [18:09:57] milimetric: Your patches lgtm. I'll find a window and we'll get it on the calendar. Anytime this week work for you mostly? [18:10:10] thx very much RainbowSprinkles [18:10:24] anytime works for me, I'm at your command RainbowSprinkles [18:10:51] * RainbowSprinkles stalks the deployment calendar [18:37:01] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3006438 (10Ocaasi_WMF) @bd808 @MSchottlender-WMF @awight Hi! We're a bit stuck but very close... [18:40:40] wikimedia/mediawiki-extensions-EventLogging#633 (wmf/1.29.0-wmf.11 - 04e3fe4 : Translation updater bot): The build has errored. [18:40:40] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.11 [18:40:40] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/199327391 [19:04:47] * elukey afk! [19:16:27] Nemo_bis: hola! yt? [19:17:38] Nemo_bis: wanted you opinion on the new wikistats frontend, please do share if you have a few mins: https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback/Round1 [19:23:07] fdans: yt? [19:23:15] yesh! [19:23:24] i have a JS question! [19:23:29] batcave? [19:23:32] ya! [19:29:06] * milimetric lunch [19:43:29] nuria: hi, thanks for writing. I watchlisted the page but I have very little time this week, will try during the weekend maybe [20:11:00] Nemo_bis: super thanks [20:24:43] !log Resubmit cassandra-coord-pageview-per-project-hourly for 2017-02-07T18:00 [20:24:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:26:43] ottomata: looking at event streams e-mail , only 1 nit: i think the questions you ask should more closely match the topics you have on talk page: https://www.mediawiki.org/wiki/Talk:EventStreams [20:27:37] ottomata: I think we are ready to go and send it today [20:27:54] joal: question if you may [20:28:10] ok awesome [20:28:11] looking [20:28:44] nuria: please :) [20:28:55] joal: I was trying to delete data from druid (tiles poc) and after disabling the dataset and issuing delete commands nothing happened [20:29:14] well, nuria in the talk page, I have an explicit 'talk topic' subject, and then the description [20:29:34] hmm, maybe I can link tehm... [20:29:51] right [20:29:52] oh cool, i can [20:30:08] nuria: from coordinator UI, no more tiles dataset [20:30:13] ottomata: linking them would make the delights of mw hardcore folks [20:30:20] joal: right, cause i "disabled" it [20:30:24] nuria: however, data is still present on hadoop [20:30:34] joal: which you need to do before deleting it [20:30:36] nuria: meaning data is not present on druid machines anymore [20:30:41] joal: data is on druid too [20:31:02] joal: mmmm.. i could disable it but not delete it [20:31:05] nuria: \i don't think so [20:31:31] nuria: they say disable in druid because data is not deleted from deep-storage (hadoop) [20:31:40] But, the data is not present on druid machines anymore [20:32:05] If you try to view it from pivot, it says: no such datasource [20:32:07] joal: check again, i renabled it [20:32:48] nuria: how's that? [20:32:50] nuria: enabled = data loaded from deep-storage onto historical machines for computation [20:32:51] haha [20:32:51] thanks [20:33:07] nuria: now, tiles-poc is being reloaded from hadoop (since you reenabled) [20:33:25] makes sense? [20:33:48] good to go from you nuria? [20:33:56] ottomata: please! [20:34:02] ottomata: ex-ci-ting! [20:34:13] ok awesome...depending on how metrics thing i'm doing goes, i mayyyy wait until tomorrow [20:34:21] but tomorrow for sure! [20:34:37] joal: ahem.. ok-i-think-so but after disabling i cannot se any task that would indicate data is being deleted [20:35:17] nuria: data is not "deleted", it's present but not manageed (meaning overwritten as needed) [20:36:11] joal: ok, so to delete completely ? [20:36:38] nuria: http://druid.io/docs/latest/ingestion/tasks.html --> kill task [20:38:26] Joal: at the end that is just a post Right? which i tried: curl -X DELETE http://localhost:8081/druid/coordinator/v1/datasources/bar/tiles-poc/2016-01-01T00:00:00.000_2017-01-01T00:00:00.000 [20:38:50] joal: maybe i got it wrong though [20:39:09] nuria: see that one too: http://druid.io/docs/latest/operations/rule-configuration.html at the bottom, Permanently deleting data [20:39:22] nuria: I think you took the wrong service [20:39:26] nuria: druid has many :) [20:40:05] nuria: 8081 is coordinator - to kill, use IndexinfService (8090) [20:40:32] joal: ah ok, let me try and if it works will document [20:40:33] And use the kill task format as defined [20:41:09] nuria: kill task is a POST with json payload [20:41:18] json as defined in the task page [20:42:27] But, nuria you also need to disable the datasource fisrt (I think this what the DELETE call did) [20:43:12] And while we are it: mforns, can I delete druid banner-impressions-hourly? [20:43:23] joal, yes sure [20:43:28] ok cool mforns [20:43:30] thank you! [20:43:36] np :) [20:44:27] joal: I do not think so, disable is done on Ui, delete call doesn't work (it is not supposed to) if datasource is not disabled [20:44:57] nuria: I don't agree - but maybe :) [20:45:50] joal: let's try it if you want, so we document properly how to delete data experiments, that is, if you are not in the middle of anything [20:46:14] nuria: currently doing it with banner-impressions-hourly [20:47:29] joal: ok [20:49:08] nuria: banner-impressions-hourly gone [20:49:19] nuria: trying with tiles [20:49:34] joal: i just issued delete cmd , wait a sec [20:49:41] arf [20:50:12] joal: no, i do not see nothing happening [20:50:26] nuria: I'll triple check something [20:50:29] give me aminute [20:50:46] nuria: just enabled tiles [20:50:54] corrd ui tells us so [20:51:05] k [20:51:15] now, please run your delete command (nothing else) [20:51:20] nuria: --^ [20:52:02] joal: ya, [20:52:04] done [20:52:17] curl -X DELETE http://localhost:8090/druid/coordinator/v1/datasources/bar/tiles-poc/2016-01-01T00:00:00.000_2017-01-01T00:00:00.000 [20:52:55] nuria: I know what happens - delete is not REST for kill tasks [20:53:02] nuria: give me a second testing [20:58:10] nuria: https://gist.github.com/jobar/076dba4ef90160fa48ac61b52c3791ef [20:59:36] joal: i see, same thing but not rest as you said [20:59:56] And the REST one is for disabling [21:03:07] joal, ok, thank you documented now here: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Druid#Delete_a_data_set_from_deep_storage [21:03:26] Thanks for the doc nuria [21:09:28] 06Analytics-Kanban, 10Wikimedia-Stream: Report number of stream connections to statsd - https://phabricator.wikimedia.org/T157492#3007210 (10Ottomata) [21:11:03] nuria: updated the doc with a warning [21:11:14] gone for now a-team, see you tomorrow ! [21:11:19] byeee! [21:11:32] ciao [21:35:14] 10Analytics, 06Performance-Team: Check if the EventLogging User Agent schema upgrade breaks any performance tool/metric - https://phabricator.wikimedia.org/T156760#3007369 (10Nuria) >Ah, because the database only fragments when the inner schema changes, not when the outer one does. We'd have two different fiel... [21:35:31] ja ok, nuria this change is including a service-template and service runner depenency update, so i want to deploy it before we announce [21:35:40] would be crappy to announce, deploy and break it :) [21:35:47] so, hopefully it'll get reviewed today and i can deploy it tomorrow [21:38:26] wow, check out druid and pivot goodies: https://imply.io/post/2016/12/01/announcing-imply-2.0.0.html [21:40:08] nice! [21:40:10] dashboards [23:19:13] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007690 (10Capt_Swing) @Reguyla that makes perfect sense. Thanks for clarifying. If some people can't use Quarry because of site-specific configurations, that should be a major issue and needs to be addressed... [23:20:36] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10Capt_Swing) p:05Triage>03High [23:22:12] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007694 (10Capt_Swing) Setting priority to high because this issue prevents people from accessing Quarry. [23:25:14] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10Halfak) I've confirmed that Quarry's OAuth consumer registration says that it is applicable to all projects -- not just Meta. So I'm not sure what's going on here. [23:30:12] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10bd808) Quarry uses OAuth for authorization (technically not really Authenication but that's a pretty nerdy detail). Since all Wikimedia accounts have been unified there are no longer per-wiki users... [23:34:18] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007726 (10Capt_Swing) @bd808 have other tools that use Meta for authorization encountered this issue? Surprises me that this is the first time we've seen the problem. It's a bit of an edge case, granted, but... [23:37:43] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007731 (10bd808) >>! In T157342#3007726, @Capt_Swing wrote: > @bd808 have other tools that use Meta for authorization encountered this issue? Surprises me that this is the first time we've seen the problem.... [23:40:41] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3007735 (10bd808)