[00:41:52] 10Analytics, 10Analytics-EventLogging, 10scap2, 10Patch-For-Review, 10Scap (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#3508611 (10Krinkle) [00:41:54] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3508610 (10Krinkle) [00:41:58] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#1589603 (10Krinkle) 05Open>03stalled [01:01:06] 10Analytics, 10Analytics-EventLogging, 10scap2, 10Patch-For-Review, 10Scap (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#1808930 (10Krinkle) >>! In T118772#3084302, @Ottomata wrote: > > [..] create an eventlogging/scap/webperf reposito... [01:01:25] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Scap (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#3508638 (10Krinkle) [01:02:31] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Scap (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#1808930 (10Krinkle) a:05Ottomata>03Krinkle [02:33:20] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3508721 (10Shizhao) 05Resolved>03Open >>! In T172379#3502782, @bearND wrote: > MCS does some filtering.... [02:50:59] (03Draft2) 10Quiddity: fix name and link of host from labs [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/370606 [03:15:19] 10Analytics, 10Performance-Team (Radar): Eventlogging client needs to support offline events - https://phabricator.wikimedia.org/T162308#3508745 (10Krinkle) [04:27:49] 10Analytics-Kanban, 10RESTBase-API, 10WMF-Legal, 10Patch-For-Review, 10Services (done): License for pageview data - https://phabricator.wikimedia.org/T170602#3508800 (10Nuria) 05Resolved>03Open [04:28:21] 10Analytics-Kanban, 10RESTBase-API, 10WMF-Legal, 10Patch-For-Review, 10Services (done): License for pageview data - https://phabricator.wikimedia.org/T170602#3436402 (10Nuria) Let's keep this open until we have verified changes deployed [07:14:35] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3508852 (10jcrespo) **I do not want to phase out anything**. We can keep using eventlogging and have a mediawiki database copy or copies for analytics-like usage for all inte... [08:51:16] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3508898 (10Liuxinyu970226) @Shizhao but where tells you that both should be the same? [10:37:36] * elukey lunch! [12:14:47] !log stop eventlogging on eventlog1001 to test kafka consumer failures [12:18:28] re-enabled [12:22:54] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3509156 (10Shizhao) @Liuxinyu970226 The views data of the same article is the same in same day, but one of... [12:30:59] helloooo team [12:42:50] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3509171 (10Liuxinyu970226) >>! In T172379#3509156, @Shizhao wrote: > @Liuxinyu970226 The views data of the... [12:47:38] mforns: o/ [12:47:44] hello elukey :] [12:47:53] mforns: when you are ok lemme know if we need to merge whitelist changes :) [13:40:16] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3509309 (10Shizhao) @Liuxinyu970226 I mean data of the same day, but lacks data of "Wiki" and "維基媒體基金會".... [13:48:29] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509313 (10elukey) >>! In T172322#3502305, @Tbayer wrote: > ...and for various other purposes. Please do not remove these tables from MySQL yet. Not planning to take a... [13:48:41] mforns: --^ [13:51:14] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: Analytics1034 eth0 negotiated speed to 100Mb/s instead of 1000Mb/s - https://phabricator.wikimedia.org/T172633#3509314 (10Cmjohnson) @elukey: first and easiest thing is to swap the ethernet cable. Since you're not servicing traffic, I will jus... [13:54:44] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: Analytics1034 eth0 negotiated speed to 100Mb/s instead of 1000Mb/s - https://phabricator.wikimedia.org/T172633#3509317 (10elukey) +1! [13:55:27] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509319 (10Halfak) [13:58:42] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509335 (10elukey) >>! In T172410#3508852, @jcrespo wrote: > The only thing I commented is that we are going to deprecate multi-source replication, so instead of one big fat... [14:20:48] !log restart varnishkafka statsv/eventlogging instances to pick up https://gerrit.wikimedia.org/r/#/c/370637/ (kafka protocol explicitly set to 0.9.0.1) [14:20:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:29:02] (03PS2) 10Milimetric: Enable all wikis to sqoop from labs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/369409 (https://phabricator.wikimedia.org/T165233) [14:29:13] (03CR) 10Milimetric: [V: 032 C: 032] Enable all wikis to sqoop from labs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/369409 (https://phabricator.wikimedia.org/T165233) (owner: 10Milimetric) [14:30:10] 10Analytics, 10Contributors-Analysis, 10DBA, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3509444 (10Milimetric) Good news, the 2017-07_private snapshot finished. I will now start the 2017-07 snapshot process, and if there ar... [14:31:21] a-team: I just verified that the 2017-07_private snapshot finished with success, I'm going to go ahead and start the 2017-07 snapshot manually. [14:31:33] mforns: how's the router? [14:31:39] \o/ [14:32:07] milimetric, goood! progressing [14:32:13] ok, great [14:32:22] lemme know if you need help [14:32:26] milimetric, I'm working now on a router-link component [14:32:50] oh cool, I'd call it something else just in case [14:32:59] that handles commits to resetState and highlight classes [14:33:00] it should be simple I think [14:33:05] sure, yea [14:33:10] makes sense [14:33:32] milimetric, there's one thing that bothers me though [14:33:41] what's that [14:34:18] before removing vue-router, python's simpleHTTPServer would accept requests to i.e. /all-projects/reading/total-pageviews [14:34:41] now, without vue-router, SimpleHTTPServer returns a 404 for that url [14:34:49] because it does not match any file [14:34:58] not sure how it worked before [14:35:21] I don't think it ever requested /all-projects/reading/total-pageviews, there's no server there [14:35:25] it requested #/all-projects/reading/total-pageviews [14:35:26] I guess when the code is productionized using say apache, this won't be aproblem [14:35:34] aha [14:35:56] wait did you make it work with /all-projects/reading/total-pageviews? instead of #/all-projects/reading/total-pageviews? [14:36:01] is #path a convention for vue? [14:36:25] it's a convention on the internet, because there are I believe event handlers for when location.hash changes [14:36:31] or something like that, I forget the details [14:36:44] well, now it doesn't use # [14:36:49] should it? [14:37:18] um... depends how well it works when someone deep-links, wouldn't they get a 404 if they visit /all-projects/reading/total-pageviews? [14:37:24] directly, not through the dashboard [14:37:59] basically people use #/all-projects/reading/total-pageviews because it allows the client side to get set up and handle the route [14:38:07] milimetric, that is my question. I wonder if when productionized like using apache this will be routed to get the root [14:38:09] dunno [14:38:31] if you can find a way to do that with the new HTML5 api, then great, but I don't think there is one [14:38:32] I see, so all paths in Wikistats2 are to be prefixed with # [14:38:36] ? [14:38:49] yeah, that's how single page apps work as far as I know, again, unless there's some new magic [14:39:08] ok, cool [14:39:16] I think this won't be difficult to do [14:39:34] oooook, thanks for the explanation! [14:39:43] getting closer [14:41:38] I apologize if I underestimated this task like I underestimate everything [14:53:11] milimetric, xD, well you certainly underestimate your capability of estimating things [14:53:33] ok, it works now :D [14:53:46] 10Analytics-Kanban, 10EventBus, 10Scap, 10Patch-For-Review, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3509512 (10elukey) [14:53:54] 10Analytics-Kanban, 10EventBus, 10Scap, 10Patch-For-Review, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10elukey) a:03elukey [14:56:33] also, it was really fun to do this [14:56:53] and I can say now I know a little bit of vue :] [15:00:56] ping fdans [15:07:09] * elukey misses fdans in team Europe [15:11:24] 10Analytics-Kanban, 10Patch-For-Review: Add QuickSurvey schemas to EventLogging white-list - https://phabricator.wikimedia.org/T172112#3486127 (10leila) @mforns I'm looking into the detail of this data and will get back to you soon. (I need to check with Florian who works with this data to make sure we don't r... [15:25:35] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509611 (10Nuria) @JKatzWMF Let us know if there is any data that can be deleted and does not need to be archived [15:28:25] ok a-team: fixed the formatting error and docs reflect the steps I took this past week to run these reconstruction jobs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Data_Lake/Administration/Edit/Pipeline#How_is_this_data_gathered:_ad-hoc_private_replicas [15:28:57] nice :) [15:29:03] mforns, milimetric : regarding the '#" [15:29:14] yep [15:29:16] milimetric, mforns : the # only works client side [15:29:25] milimetric, mforns : teh server doesn't see it [15:29:30] right, of course [15:29:33] sure [15:30:05] mforns: so any server (apache or other) will behave the same given teh same url [15:30:34] yes [15:31:54] mforns: ok so no issue on apache versus anything else [15:31:59] nuria_, no no [15:32:09] it was purely a client-side thing [15:32:19] I forgot to add the # in the paths [15:32:31] but now it's working fine [15:32:39] with # [16:09:36] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3509744 (10Nuria) @Shizao: Some pageviews that "appear" to be from users are actually from bots, some of th... [16:12:49] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509765 (10Nuria) Ok, seems that this ticket can be closed as it talks about work that is tracked on tickets and projects elsewhere. [16:12:55] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509766 (10Nuria) 05Open>03Resolved [16:14:10] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509773 (10Halfak) Is switching analytics-store from multi-source to multi-instance tracked elsewhere? Is there a task for identifying the user-implications of making the sw... [16:16:52] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3509774 (10jcrespo) It is a Tecnology goal, it was already discussed by all managers: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q1 [16:23:47] 10Analytics-Kanban, 10RESTBase-API, 10WMF-Legal, 10Patch-For-Review, 10Services (done): License for pageview data - https://phabricator.wikimedia.org/T170602#3509783 (10Nuria) @mobrovac Please ping on ticket when deployment happens [16:43:12] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Write generic certificate management software for use with Puppet and Self Signing CAs. - https://phabricator.wikimedia.org/T166167#3509832 (10Jgreen) [16:47:46] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509840 (10JKatzWMF) @elukey @nuria my query is either failing or returning no results due to an error in the logic. My querying skills may have atrophied or the table may have... [16:50:19] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509845 (10JKatzWMF) pinging @Halfak, @Neil_P._Quinn_WMF @Jdforrester-WMF: Can we delete this table: PageContentSaveComplete_5588433_1542324? It is 457.22 gb and as impossible... [16:53:15] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509863 (10elukey) >>! In T172322#3509845, @JKatzWMF wrote: > pinging @Halfak, @Neil_P._Quinn_WMF @Jdforrester-WMF: Can we delete this table: PageContentSaveComplete_5588433_15... [16:53:22] milimetric: how can i see crontab of hdfs on stat1003? [16:53:33] https://www.irccloud.com/pastebin/zSnlPYFK/ [16:53:43] sudo -u hdfs crontab -l [16:54:10] mmmm [16:54:11] yeah [16:54:12] sorry [16:54:31] elukey: right [16:54:36] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3509866 (10JKatzWMF) ^ oh great! the next one on the list is mediaviewer and this needs to stay for the time being. A new multimedia PM should be coming on board soon and I th... [16:54:43] elukey: that is what i though [16:54:47] I checked the paste after sorry [16:54:52] elukey: does that work on 1003? [16:54:56] elukey: ah np, [16:55:09] so stat1003 does not have access to hdfs now that I think about it [16:55:15] so hdfs should not be there [16:55:20] milimetric: if i understand right the scoop job for mw edit reconstruction is on 1003 [16:55:31] elukey: what user is it under? [16:56:06] elukey: or wait, maybe now it is on 1005 [16:56:11] elukey: one sec [16:56:41] elukey: no it is not [16:57:11] elukey: then it must be root on 1003 who executes the sqoop job [16:57:19] it is probably on analytics1003 [16:57:50] yep :) [16:57:57] elukey@analytics1003:~$ sudo -u hdfs crontab -l | grep sqoop [16:57:57] # Puppet Name: refinery-sqoop-mediawiki [16:58:06] nuria_: --^ [16:58:15] elukey: ya, must be 1006 then [16:58:32] ?? [16:59:10] elukey: sorry, i see cron on puppet too [16:59:22] elukey: it says user is hdfs [16:59:30] elukey: but .. on what machine? [17:03:41] wikimedia/mediawiki-extensions-EventLogging#678 (wmf/1.30.0-wmf.13 - dd54cdd : Antoine Musso): The build has errored. [17:03:41] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.30.0-wmf.13 [17:03:41] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/262333964 [17:05:13] nuria_: what is the name of the cron? [17:05:26] elukey: refinery-sqoop-mediawiki [17:05:50] nuria_: I pasted it above, analytics1003 :) [17:06:04] elukey: right, me no comprendou [17:06:17] elukey: user => 'hdfs', [17:06:29] elukey: but as you said hdfs is not on 1003 [17:06:36] cc milimetric maybe for help [17:06:44] hdfs is not on stat1003, not analytics1003 [17:07:00] elukey: ahahah [17:07:32] elukey: ok ,argh, dislexia, Sorry [17:07:40] all sorted? [17:07:48] sorry I missed the previous ping, what's the confusion? [17:08:17] milimetric: ya, sorry [17:08:18] analytics1003, # Puppet Name: refinery-sqoop-mediawiki [17:08:58] if there's any confusion around this process, please let me know, I'm not sure what specifically to help document more [17:09:28] milimetric: no, my faul [17:09:37] read analytics1003 and [17:09:47] milimetric: OF Course i wenbt to look for things in stat1003 [17:09:54] milimetric: duh [17:10:04] k, cool [17:10:27] milimetric: should i scoop then PagecontentSavecomplete from that machine too? [17:10:33] milimetric: os it doesn't matter? [17:11:01] *or it doesn't matter [17:12:53] nuria_: I do the same thing once in a while, and then I realize I am on the wrong host :D [17:25:08] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Analytics Kafka cluster causing timeouts to Varnishkafka since July 28th - https://phabricator.wikimedia.org/T172681#3509923 (10elukey) All the varnishkafkas are now forced (via lidbrdkafka settings) to communicate using the kafka `0.9.0.... [17:37:27] elukey: what do we feel is a good directory to backup Pagecontentsavecomplete on hdfs? [17:37:41] cc milimetric [17:38:08] no idea :( [17:38:43] elukey: ok, i will pick one and we can move things if we do not like it [17:40:54] elukey, milimetric , mforns How about /wmf/data/archive/eventlogging/Table_name [17:41:15] sounds good [17:41:31] nuria_, I like it too [17:41:34] but maybe inside have it partitioned by schema revision? [17:41:36] not sure... [17:41:48] +1 [17:43:31] milimetric: that seems fancy for the "get tables out of the way" situation [17:43:45] sall good [17:44:25] milimetric: maybe is worth thinking what will we do for our next dump of this table though, same schema [17:44:29] same table name [17:45:01] cc mforns elukey [17:46:44] nuria_, will the files be partitioned by time? i.e. in hourly folders? [17:47:00] mforns: no, these are just plain dumps of tables [17:47:16] nuria_, with camus? [17:47:24] mforns: no, with sqoop [17:47:41] sorry sqoop yes, ok [17:47:52] mforns: we are just moving the data from mysql to hdfs for (at least) pagecontentsavecomplete [17:48:00] yea [17:48:06] mforns: other tables might need fancier formatting [17:48:33] aha, to be queryable [17:50:06] mforns: right [17:52:53] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3510014 (10GWicke) p:05Triage>03Normal [17:55:54] elukey: how do i do to kill a job such us this one on the cluster: https://yarn.wikimedia.org/cluster/app/application_1498042433999_173847 [17:56:06] elukey: i.e. to re-start my import? [17:59:03] elukey: i used hadoop job -kill , hopefully that is ok [18:15:57] mforns, milimetric , elukey : so we are in the same page that when i scoop Pagecontentsavecomplete we will just have a dump of table in text based format, right? [18:16:20] this type of dump will not work to back up data for redaing as dat woudl need to be in a table format [18:17:11] nuria_: sorry just seen the msgs, puppet shower errors in #operations, had to deal with it [18:17:22] nuria_, will the file be readable by Andrew's refinery? [18:17:35] I guess not, but is that important? [18:18:00] mforns: it is text but not a hive table, it can be imported into a hive table but then it will need a partition scheme [18:18:02] it might also be good that we use parquet [18:18:06] I mean, which program will be able to read that data? [18:18:16] mforns: hdfs -ls [18:18:18] hive not, because it is a biiig file without partitions [18:19:13] my question is, in case someone wants to read or recover that data, how are they going to do that? [18:20:26] mforns: if it is text it can be recover with hdfs dfs -get /path/ [18:20:57] mforns: if we want to allow query-able access we need to create a hive table with partitions and such, i can try to experiment with that [18:21:18] not sure if that is needed though [18:21:26] but just wanted to make sure it is usable [18:22:10] hdfs dfs -get /path will transfer a lot of data no? [18:23:16] nuria_, is there any way we can import it now as a file and afterwards partition it if necessary? [18:23:33] mforns: it is imported as set of files [18:23:38] mforns: like [18:23:38] https://www.irccloud.com/pastebin/4idatUgq/ [18:23:40] aaaaaaaaaah [18:23:51] mforns: that is an example of EMPLOYEES table [18:24:05] I thought it was a file [18:24:10] ok [18:24:14] mforns: but still , not query-able with sql [18:24:49] then I guess it's totally possible to process it afterwards with a spark job or similar to partition it [18:24:54] if we need it no? [18:26:49] mforns: yes, but maybe is worth trying to import this as hive tables with some partition scheme [18:27:37] it depends on how likely it is that people might want to look at that data in a query fashion, because via spark it can always be processed. [18:27:40] nuria_, ^ [18:27:45] mforns: but it is not obvious what column to use to partition either [18:28:12] checking analytics1055 (page) [18:28:29] mforns: for pageContentSaveComplete i am not super worried cause mostly we want that table out of the way [18:28:33] ah lovely, disk broken [18:28:39] elukey: argh [18:28:41] nuria_, yea.. ideally it would be year, month and day, extracted from the timestamp [18:28:58] :[ [18:29:42] nuria_ / mforns, there doesn't need to be a partition scheme to put a hive table on top of it [18:29:46] it's only a convenience [18:29:58] we can put a hive table on top of flat files [18:30:14] they just have to have a standard format [18:30:28] milimetric: yes, but if we do import #2 of same table without a partition scheme [18:30:42] milimetric: there is no way to "add" that dat ato original table [18:30:49] milimetric: it could overwrite it [18:30:51] you can overwrite, assuming it's not used every day [18:30:52] yea [18:30:52] milimetric, aha, but the data is big, if it has no partitions... actually it's not soo big for hive [18:30:59] milimetric: but we cannot overwrite [18:31:03] milimetric: this is abackup [18:31:48] right, you can "overwrite" as in, move to _backup, write the new one, delete the backup [18:31:52] nuria_, milimetric, we can add the timestamp of the dump as part of the table name [18:32:02] milimetric: no, cause we are deleting data from EL [18:32:14] I understand [18:32:32] milimetric: k, once imported data exists in hadoop only [18:32:55] mforns: we can make each dump its own partition [18:33:02] 'a la mw history dump' [18:33:05] that's fine, as long as you don't have duplicates, you can just add more files into the same folder and the table will just pick them up [18:33:25] you just have to keep track of what the max(timestamp) or max(id) or something is, and give that to sqoop next time you import [18:33:39] aha [18:33:48] milimetric: no need cause table will only have the data you want [18:33:56] milimetric: the older one has been deleted [18:34:01] basically, don't worry about it now, worry about it if people actually use it or if we need to migrate it to hive permanentely [18:34:45] not always nuria_, you can have duplicates later if you keep importing [18:34:54] and you import faster than you truncate old data [18:35:08] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Broken disk on analytics1055 - https://phabricator.wikimedia.org/T172808#3510118 (10elukey) [18:35:16] milimetric: no, wait, this is teh context of freeing space [18:35:19] there you go --^ [18:35:22] milimetric: we will import data [18:35:28] so analytics1055 down [18:35:32] kaput for the moment [18:35:33] milimetric: and delete table on mysql [18:35:42] elukey: you want me to look at restarting those jobs that failed so you're free to go? [18:35:50] milimetric: thus next import will not contain any data we have already imported [18:35:56] nuria_: yes, but that schema is still generating events, yes [18:36:03] so you import now, delete, keeps generating [18:36:07] import again, no dupes [18:36:12] delete stuff older than 90 days [18:36:14] ongoing [18:36:18] then it keeps generating [18:36:21] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3510135 (10fdans) [18:36:22] then you import again [18:36:28] that's when you might get dupes [18:36:28] milimetric: we are not deleting any data here [18:36:35] milimetric: we are keeping it all [18:36:41] uh... I think we're really talking past each other here [18:36:45] milimetric: doing it now, but thanks! [18:36:59] I understand nuria, we're deleting it on mysql dbs, yes? [18:37:11] milimetric: right, you are talking about eventlogging workflow, [18:37:27] i'm talking about ongoing imports, the reason we're talking about partitions [18:37:30] nuria_, but another thought, there will be a gap between the end of the sqoop job and the deletion of the EL table in analytics dbs, that data will be lost, no? [18:37:54] mforns: not if when we import we stop el [18:38:07] but import is going to take a while no? [18:38:18] as I said, you don't have to worry about this [18:38:20] milimetric: not sure, i am timing a fake import now [18:38:30] cc mforns [18:38:31] k [18:38:33] next time you want to import, just select max(something) from the table in hive and pass that to sqoop [18:38:54] maybe batcave? [18:38:57] k, sure [18:39:01] !log restart projectview-hourly-wf-2017-8-8-14, pageview-druid-hourly-wf-2017-8-8-14, pageview-hourly-wf-2017-8-8-14 via Hue (analytics1055 disk failure) [18:39:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:39:06] milimetric: done! [18:39:15] elukey: thanks for restarting those [18:39:36] in the cave nuria_ & mforns [18:39:50] ok! [18:40:05] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Broken disk on analytics1055 - https://phabricator.wikimedia.org/T172808#3510161 (10elukey) [18:40:26] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Degraded RAID on analytics1055 - https://phabricator.wikimedia.org/T172809#3510136 (10elukey) [18:41:42] going offline team! [18:41:46] talk with you tomorrow :) [19:08:32] milimetric: did you started the labs mw edit reconstruction job? [19:17:41] 10Analytics-Kanban, 10Analytics-Wikistats: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3510227 (10Nuria) @Slaporte : can you pleae take a look at footer? https://stats.wikimedia.org/v2/#/all-projects [19:19:44] nuria_: hm, sqoop job has been stuck for a while [19:21:25] 10Analytics-Kanban, 10Analytics-Wikistats: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3510261 (10Slaporte) This looks good. [19:23:46] nuria_: it looks like sqoop asked hadoop for a job and it's not getting resources [19:24:12] probably because of the crash/restart I would guess [19:24:28] normally I might put it in the priority queue, but I think in this case maybe it's better to let it wait [19:24:38] because it's not crucial by any means and it's actually more of a pain to restart it [19:25:30] milimetric: ya, i did kill my original job (that was trying to scoop 400G) [19:25:39] milimetric: and set one up for about 60G [19:26:39] that still has not received resources [19:26:51] milimetric: but i will just wait couple hours while i do something else [19:27:23] yeah, I think the cluster is a little stuck, but I don't know enough to kick it [19:32:58] 10Analytics-Kanban: Add Licensing info to analytics.wikimedia.org - https://phabricator.wikimedia.org/T172814#3510280 (10Nuria) [19:33:39] 10Analytics-Kanban: Add Licensing info to analytics.wikimedia.org - https://phabricator.wikimedia.org/T172814#3510293 (10Nuria) [19:34:22] milimetric: ok, 60G task failed with: "Cannot run program "mysqldump": error=2, No such file or directory" , sounds familiar? [19:38:19] nuria_: sounds like the options of sqoop you used needed mysqldump, and it's not available on the hadoop nodes [19:38:29] maybe that's another reason to go with --asavrodatafile [19:38:35] (not sure if that's how you spell it) [19:38:40] milimetric: ahahammm [19:46:58] yeah, that's the mind-bending thing about sqoop, it has to link three separate systems in harmony - your local, hadoop, and the remote source [19:53:31] 10Analytics: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3510348 (10Nuria) [19:53:33] 10Analytics-Kanban, 10Patch-For-Review: Upgrade Druid to 0.9.2 as a temporary measure - https://phabricator.wikimedia.org/T170590#3510347 (10Nuria) 05Open>03Resolved [19:53:42] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review, 10User-Elukey: Update Varnishkafka to support TLS encryption/authentication - https://phabricator.wikimedia.org/T165736#3510349 (10Nuria) 05Open>03Resolved [19:54:23] 10Analytics-Cluster, 10Analytics-Kanban: Provision new Kafka cluster(s) with security features - https://phabricator.wikimedia.org/T152015#3510351 (10Nuria) [19:54:25] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Understand Kafka ACLs and figure out what ACLs we want for production topics - https://phabricator.wikimedia.org/T167304#3510350 (10Nuria) 05Open>03Resolved [19:54:41] 10Analytics-Kanban, 10Analytics-Wikistats, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3510352 (10Nuria) 05Open>03Resolved [20:00:12] yay, my job kept going [21:11:30] 10Analytics-Kanban, 10RESTBase-API, 10WMF-Legal, 10Patch-For-Review, 10Services (done): License for pageview data - https://phabricator.wikimedia.org/T170602#3510522 (10Pchelolo) 05Open>03Resolved Deployed. Resolving. [21:27:12] this job is retrying constantly [21:59:00] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3510763 (10Pchelolo) [23:16:21] (03PS1) 10Nuria: Adding License to analytics.wikimedia.org [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/370761 (https://phabricator.wikimedia.org/T172814) [23:17:25] (03CR) 10Nuria: [V: 032 C: 032] "Self merging, modified license text at the request of legal" [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/370761 (https://phabricator.wikimedia.org/T172814) (owner: 10Nuria) [23:36:46] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3510972 (10Nuria) The command time sudo -u hdfs sqoop import --as-avrodatafile --password-file '/user/hdfs/mysql-analytics-research-client-pw.txt' --usern... [23:53:54] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3510978 (10Nuria) [23:54:21] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3440380 (10Nuria)