[00:13:23] 10Analytics, 10Analytics-General-or-Unknown: X-Analytics header is "php=zend;php=zend" instead of "php=zend" on bits for some requests - https://phabricator.wikimedia.org/T72463#3135868 (10Krinkle) 05Open>03declined Closing since bits.wikimedia.org no longer exists (T107430). Also, as far as I know, we no... [06:31:07] morning! [06:31:20] last occurrence of DiskErrorException on an1044 was 2017-03-27 14:06:02 [06:31:23] \o/ [06:31:53] so apparently a simple restart of the daemon fixed the issue (even if I changed some perms on the main data dir but not only on 1044) [08:18:16] 10Analytics, 10Analytics-Cluster, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3136346 (10elukey) We found a weird regression only on analytics1044, causing sporadic job failures: ``` 2017-03-27 14:06:02... [08:32:48] Awesome elukey :) [09:18:06] legacy pagecounts are deployed in restbase :D [09:18:29] mforns: yes, since yesterday evening :) [09:18:32] Yay ! [09:18:37] \o/ finally.. [09:18:58] Congrats mforns :) New dataz ! [09:19:08] yea... :] [09:20:00] nice work! [09:20:16] :] [09:30:14] niiiiice mforns! [09:30:51] :] cool that you can test the pageviews.js and dashiki stuff now [09:41:39] (03PS13) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [09:43:09] (03CR) 10Mforns: "> re table var names, sounds ok to me! Just make sure that there is" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) (owner: 10Mforns) [09:44:44] (03PS14) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [10:18:08] a-team: I just merged the change to switch hue.w.o from analytics1027 to thorium [10:18:19] let me know if you see any weirdnesss [10:23:52] nice I can see proxy-server/200 on thorium for hue [10:23:54] \o/ [10:31:25] 10Analytics, 06DC-Ops, 06Operations, 10ops-eqiad: Decom/Reclaim analytics1027 - https://phabricator.wikimedia.org/T161597#3136552 (10elukey) [10:35:44] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Move away Hue and Camus (and other crons) from analytics1027 - https://phabricator.wikimedia.org/T159527#3136578 (10elukey) Created https://phabricator.wikimedia.org/T161597 to decommission analytics1027. @Ottomata anything left to do? [10:51:08] elukey: for me issue so far [10:52:09] no issue? :D [10:54:37] 06Analytics-Kanban, 06Operations, 10Traffic, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3136586 (10elukey) The traffic is definitely decreased a lot from last week, but I am still seeing some 503s (way more than before). I a... [11:00:38] on hue :) elukey [11:01:20] joal: oh yes but you said "issue so far" and at first reading I thought something was exploding :) [11:01:50] elukey: Ho my bad ! I meant NO issue so far - But the important part of it, I forgot ! [11:02:26] excuse me elukey for the typo-of-the-day [11:03:00] no worries! I just found a real problem, snap [11:03:09] Mwarf :( [11:03:47] does hue.wikimedia.org alone works for you joal ? [11:04:37] actually now I got the reload popup never stopping [11:06:10] elukey: removed cookies, now error [11:06:29] joal: but https://hue.wikimedia.org/oozie/list_oozie_workflows/ works right? [11:06:53] ah no that one not works too.. the Location header is wrong [11:07:41] elukey: I thought it was working because the refresh popup goes out after relaoding, but actually it wasn't reloaded [11:07:49] It's a bit misleading [11:08:15] When I try to login elukey I got a 403 CSRF error. [11:10:43] the Location header in the response has localhost instead of the proper link, really weird [11:10:59] :( [11:11:36] let me try one thing [11:14:45] joal: ok so I fixed the Location header issue [11:15:04] now hue.wikimedia.org should not redirect to localhost:888 [11:15:18] but I can see the CSRF error [11:15:19] mmmmm [11:16:02] Referer checking failed - https://hue.wikimedia.org/accounts/login/?next=/oozie/list_oozie_coordinators/ does not match https://localhost:8888/ [11:16:05] -.- [11:16:21] ;-( [11:16:34] so this is a issue using Hue behind a proxy [11:16:40] it surely needs headers [11:19:14] joal: can you retry now please? [11:19:40] elukey: SUCCESS ! [11:19:43] \o/ [11:19:57] ok found the second problem, it was the Host header sent to hue [11:20:07] going to send a patch in a bit [11:28:20] taking a break a-team [11:28:26] k :] [11:46:38] joal: just fixed the apache config, all good (it seems) [11:46:49] going to lunch, let me know if something comes up! [12:40:51] 10Analytics-Tech-community-metrics: Maniphest: Parser does not split projects by comma separator? - https://phabricator.wikimedia.org/T161519#3136740 (10Aklapper) 05Open>03Resolved a:03Albertinisg @Albertinisg: Thanks a lot for the quick fix! (I'm wondering if {T161241} would also be that easy - I guess n... [12:51:17] 10Analytics-Tech-community-metrics: Maniphest: Parser does not split projects by comma separator? - https://phabricator.wikimedia.org/T161519#3136753 (10Albertinisg) I will check it out! [13:26:35] hiaaa [13:26:40] elukey: hi, have you turned off hue on an27 yet? [13:26:40] o/ [13:26:46] nope [13:26:57] code review ready but haven't touched anything [13:27:07] aye, hue.wm.org feels really slow to me atm... [13:27:09] want to compare [13:27:21] ahhh ok so you meant if I switched to thorium? [13:27:29] if so yes I did it this morning [13:27:29] naw, it is switched arleady, right? [13:27:31] right [13:27:35] okok [13:27:38] :) [13:28:14] anything in particular that is super slow? [13:28:30] hmm, no i think its fine actually, might have just been me for a minute [13:28:35] now i'm clicking around and it seems fine [13:29:39] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Move away Hue and Camus (and other crons) from analytics1027 - https://phabricator.wikimedia.org/T159527#3136800 (10Ottomata) Nope! Looks good to me! [13:29:46] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Move away Hue and Camus (and other crons) from analytics1027 - https://phabricator.wikimedia.org/T159527#3136801 (10Ottomata) [13:31:52] super! I had to make some adjustments to the apache config to please hue [13:32:47] nice mforns, good work, I think a lot of people will be super happy, you should write the announcement to analytics-l [13:33:28] milimetric, thanks, sure, I think I'll write it, but first I'd like to add the mobile counts [13:33:46] saw that [13:33:48] and also vet the data in dashiki in comparison to pageviews [13:34:00] cool, makes sense [13:34:09] joal: hiiii yt? [13:34:13] hi ottomata ! [13:34:28] I'm here :) [13:34:31] hiya! did you see my email from friday about python deps stuff? [13:34:35] I did ! [13:34:43] let's discuss in da cave for a minute ? [13:34:52] ya k [13:48:55] (03PS1) 10Mforns: Use both projectcounts raw and all sites to load cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345144 (https://phabricator.wikimedia.org/T161494) [13:49:31] (03CR) 10Mforns: [C: 04-1] "Still WIP, needs to be tested." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345144 (https://phabricator.wikimedia.org/T161494) (owner: 10Mforns) [13:57:50] nuria: mforns PR merged in pageviews.js \o/ [13:58:18] fdans, \\\\\\o/////// [13:58:33] you're google-proof now fdans :] [13:59:07] mforns: haha he made a few changes, but at least he didn't close and reopen!! [13:59:18] awesome :] [14:02:18] Hi halfak [14:02:33] halfak: Have you had your coffee already? [14:09:29] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (designing), 15User-mobrovac: Puppetize event schema topic configuration - https://phabricator.wikimedia.org/T161027#3136918 (10Ottomata) For Vagrant at least, I don't think we really need the topic config. We could make EventBus default to not enf... [14:14:41] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (designing), 15User-mobrovac: Puppetize event schema topic configuration - https://phabricator.wikimedia.org/T161027#3136923 (10Pchelolo) +1 to @Ottomata [14:14:51] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3136924 (10Ottomata) [14:15:26] 10Analytics, 10Analytics-Cluster, 06DC-Ops, 06Operations, 10ops-eqiad: Analytics1028 hdfs daemon died because of disk errors - https://phabricator.wikimedia.org/T159632#3136925 (10Cmjohnson) Swapped the disk out with a spare on-site. The server is still under warranty so requested a new disk to be sent t... [14:15:50] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (designing), 15User-mobrovac: Puppetize event schema topic configuration - https://phabricator.wikimedia.org/T161027#3136926 (10Ottomata) > We could maintain a dev/example topic-config.yaml in event-schemas, Actually, if we do this, maybe it would b... [14:25:09] RECOVERY - Hadoop DataNode on analytics1028 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [14:25:24] (03PS2) 10Mforns: Use both projectcounts raw and all sites to load cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345144 (https://phabricator.wikimedia.org/T161494) [14:28:39] RECOVERY - Hadoop NodeManager on analytics1028 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [14:30:03] !log analytics1028 back serving traffic - T159632 [14:30:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:04] T159632: Analytics1028 hdfs daemon died because of disk errors - https://phabricator.wikimedia.org/T159632 [14:30:11] joal,ottomata --^ [14:30:30] Yay elukey ! [14:30:39] The return of an1028! [14:31:11] 10Analytics, 10Analytics-Cluster, 06DC-Ops, 06Operations, 10ops-eqiad: Analytics1028 hdfs daemon died because of disk errors - https://phabricator.wikimedia.org/T159632#3136970 (10elukey) 05Open>03Resolved a:03elukey [14:32:30] yeehaw! [14:32:46] thanks elukey! [14:35:04] ottomata: whenever you have time can you check stat1002's puppet run? [14:35:09] it fails in git pull.. [14:38:08] lloking [14:39:16] why?! haha why woul that just break!? [14:43:42] I checked very quickly and didn't find any explanation :D [14:48:54] elukey: it looks like there was some discrepency between working copy master and origin/master [14:48:59] git pull wanted to do a merge commit [14:49:09] i deleted master and re-checked it out from origin/master [14:49:10] and then puppet ran [14:49:18] ¯\_(ツ)_/¯ [14:50:58] fdans: hello. [14:51:06] hi! [14:51:29] fdans: you are probably going to need to modify the config to test your changes to reportcard. You can modify : https://meta.wikimedia.org/wiki/Config:Dashiki:Sample/tabs [14:51:52] ottomata: (╯°□°）╯︵ ┻━┻ [14:52:08] haha [14:52:14] which default for tabs layout (will document on README) , if you go to http://localhost:port/src that config will be source by default [14:52:16] nuria: I wanted to avoid creating new configs in metawiki, so I'm overriding the config locally [14:52:17] whoa elukey CHILL OUT [14:52:18] its ok [14:52:19] ! [14:52:34] fdans: that works too, the sample one exists for that pupose [14:53:08] nuria: oh cool, I'll edit it later when I got tabs working locally [14:53:42] ottomata: ahahhaha you can flip the table quietly! It is a good way to remove the stress [14:53:50] hahahaah [14:55:21] this is elukey: [14:55:22] (•_•) [14:55:22] ( •_•)>⌐■-■ [14:55:22] (⌐■_■) [14:56:28] ahhahaah [14:56:30] hahah [14:56:34] hahahaa [14:56:36] that's a good one! [15:17:28] 06Analytics-Kanban: Document and publicize AQS legacy page counts endpoint - https://phabricator.wikimedia.org/T159959#3137164 (10Nuria) [15:21:10] joal: I gotta take a bathroom break, sorry will it be too late after retro to chat sqoop? [15:21:14] I'll think about it [15:21:19] sure [15:21:28] milimetric: no problem :) [15:25:15] 10Analytics-EventLogging, 06Analytics-Kanban, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3137202 (10Nuria) mysql:research@analytics-store.eqiad.wmnet [log]> show tables like 'Image... [15:25:22] 10Analytics-EventLogging, 06Analytics-Kanban, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3137203 (10Nuria) 05Open>03Resolved [15:25:36] 06Analytics-Kanban: Investigate duplicate EventLogging rows - https://phabricator.wikimedia.org/T142667#3137205 (10Nuria) 05Open>03Resolved [15:26:54] 06Analytics-Kanban: Document and publicize AQS legacy page counts endpoint - https://phabricator.wikimedia.org/T159959#3137212 (10Nuria) [15:26:56] 06Analytics-Kanban: Kill limn1 - https://phabricator.wikimedia.org/T146308#3137211 (10Nuria) [15:56:10] 10Analytics: Improve Oozie error emails for testing - https://phabricator.wikimedia.org/T161619#3137366 (10Milimetric) [15:58:22] 10Analytics, 10Analytics-Cluster, 06Operations, 10ops-eqiad, 15User-Elukey: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#3137372 (10Cmjohnson) @elukey I have the thermal paste....want to plan for this on Thursday morning (my morning)? [16:01:34] 10Analytics, 10Analytics-Cluster, 06Operations, 10ops-eqiad, 15User-Elukey: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#3137378 (10elukey) >>! In T132256#3137372, @Cmjohnson wrote: > @elukey I have the thermal paste....want to plan for this on Thursday morni... [16:19:21] ottomata: I've read quickly on kombucha - I have done something more or less similar: Kefir [16:19:34] oh ya! [16:19:43] oh right, eyah [16:19:48] cause its got that floating brain thing in it [16:19:58] i tried doing kefir once [16:20:03] it was always too gross for me :) [16:20:03] ottomata: the floating thing is weird for sure ! [16:20:08] :) [16:20:13] but, kombucha can be gross too [16:20:14] Kombucha is better? [16:20:20] ya, at the right time [16:20:29] i get about a 5 day window where it is amazing [16:20:34] fizzy, not too sour, not too sweet [16:20:35] Right, seems like the traditionel problem with fermented stuff [16:20:38] ya [16:20:55] i really love it in the summer [16:21:02] i drink so many cups of iced kombucha [16:21:04] yeah, it's really good :) [16:35:22] fdans: wait you wanted to talk to me too? [16:35:25] * elukey off! o/ [16:35:33] have a good night, luca [16:36:09] milimetric: I'm good for the moment, you off? [16:36:26] no, I'm here for another 2 hours and then moving to the airport and flying back [16:40:14] 10Analytics, 06Analytics-Kanban: Update pivot to latest version - https://phabricator.wikimedia.org/T161630#3137628 (10Nuria) [17:11:59] Morning analytics! [17:15:57] AndyRussG: hello [17:16:09] nuria: :) [17:29:42] wikimedia/mediawiki-extensions-EventLogging#641 (wmf/1.29.0-wmf.18 - 838abb7 : Translation updater bot): The build has errored. [17:29:42] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.18 [17:29:42] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/216030149 [17:30:09] Just a heads-up--we're looking into a consistent system for saving banner history data, and also a good place to put the data that we've saved in an ad-hoc manner. (Previously cleared with legal..) Possibly somewhere in Hive might be good... Just gonna start an e-mail thread and a Phab task for this... [17:30:45] Whom should I cc? ottomata? Many thx in advance as always :) [17:31:15] (ottomata also thx again for ur help trying to restore that file...) [17:32:51] AndyRussG: ya, cc me, maybe joal too [17:33:13] woiuld prefer phab task :) [17:33:16] to email [17:35:08] ottomata: K gotcha :) I'll mostly write in the Phab task and just send an e-mail with a link to be sure all interested parties see it, if that's OK :) thx! [17:35:23] k [17:35:28] cool [17:39:44] ottomata: yt [17:40:22] joal: weird [17:40:23] java.io.FileNotFoundException: File /tmp/unittest/refinery-source/refinery-job/TestSubgraphPartitioner/c06baa39-dae1-438e-b765-c6dab6638bc8 does not exist [17:40:26] that's from the refinery guard run [17:40:29] on stat1002 [17:40:31] nuria: ya hey [17:41:37] ottomata: do you think we could have time to work on refreshing kafka machines by the end of next quarter? looks like we might get that hardware earlier [17:42:35] whoa [17:42:36] hm [17:42:49] i think we could order and start them, but we've barely thought about the process of switching [17:42:54] not sure how much security features we want to do [17:43:01] not totally sure how we are going to mirror and then switch [17:43:09] ottomata: but refresh doesn't need upgrade , or does it? [17:43:10] but, we could work on it, [17:43:13] yes [17:43:13] it does [17:43:29] we're taking the refresh opportunity to do the upgrade [17:43:38] we want to make a brand new cluster [17:43:49] so, we'll need them both running for a period of time [17:44:14] ottomata: i see, so it seems that is easy for us to order those now, we will do that so by end of quarter ( if hw is here) we can devote some time [17:44:46] yeah, doen't hurt to have it early [17:45:03] i do'nt think we can promise to have it totally switched by then [17:45:18] but we can start work on it [17:45:19] for sure [17:45:28] uhhh, not remembering all the other goals we talked about [17:45:29] last time it took us 3months to get hw so if we order now we can start working on this end of quarter, next quarter [17:45:31] are those on mw.org? [17:45:50] ottomata: goals for this quarter are on mw but it doesn't include naything kafka [17:45:54] *anything [17:46:11] or maybe a small item that was hardware independent, let me see [17:46:38] ottomata: "preparation work for upgrading kafka and enabling security features in Q1" [17:46:41] More scalable eventlogging solution when it comes to throughput and reliability, stat1002/3 replacement, preparation work for upgrading kafka [17:46:46] so, sure! [17:46:55] why not, we can include getting the hardware under prep work [17:47:00] we'll just have to prioritize between those thigns for me [17:47:31] ok, faidon the boss will contact you to place orders, if experience is a guide it is likely we do not get hw for 2 more months [17:48:25] cool [17:48:49] cool, he just did [17:48:51] gonna file hw req [17:48:52] now [17:51:45] ottomata: gracias seniorrr [17:52:41] 10Analytics, 10Fundraising-Backlog: Storage for banner history data - https://phabricator.wikimedia.org/T161635#3137911 (10AndyRussG) [17:53:12] 10Analytics, 10Analytics-EventLogging: Turn on consumer that produces to kafka again - https://phabricator.wikimedia.org/T68528#3137930 (10Ottomata) 05Open>03Invalid Old! [17:58:39] 10Analytics, 10Analytics-Cluster: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#3137952 (10Ottomata) [17:58:42] 10Analytics, 10Fundraising-Backlog: Storage for banner history data - https://phabricator.wikimedia.org/T161635#3137911 (10Nuria) @AndyRussG: can we look at the data to make sure it is safe to retain, the risk normally comes from cross checking datasets and evaluating that risks is something we have to do befo... [17:59:48] 10Analytics, 10Analytics-Cluster: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#2835361 (10Ottomata) I just heard that there is some budget to start provisioning these clusters sooner rather than later! :) [18:03:41] elukey: , you still there? [18:07:44] milimetric: npm is such a total major pain: https://github.com/mapbox/mapnik-swoop/issues/16 [18:08:40] shortest rant ever [18:09:30] I think npm and the node ecosystem is a million times better than ruby [18:09:38] and it's undeniably more agile than debian [18:09:52] I think if we could fuse npm with debian we'd all rest easier [18:12:07] 10Analytics, 10Analytics-Cluster: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#3138003 (10Ottomata) [18:13:15] milimetric: i disagree, the very granular node dependencies make everything much to cumbersome [18:13:27] 10Analytics, 10Analytics-Cluster: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#2835361 (10Ottomata) [18:13:58] milimetric: dashiki, which is a tiny tiny app, has 1000 dependency line tree [18:14:02] nuria: sure, but what's the latest version of <> available on debian [18:14:04] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: 6 Nodes for Kafka refresh/upgrade - https://phabricator.wikimedia.org/T161636#3138026 (10Ottomata) [18:14:11] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: CODFW: 6 Nodes for Kafka refresh/upgrade - https://phabricator.wikimedia.org/T161637#3138028 (10Ottomata) [18:14:27] the slowness of debian might be nice but it still holds progress back a lot [18:14:36] nuria: omg, I have a great article for you [18:15:05] nuria: sorry, I shouldn't have said anything and just responded with this: https://medium.com/friendship-dot-js/i-peeked-into-my-node-modules-directory-and-you-wont-believe-what-happened-next-b89f63d21558 [18:15:11] you have to read that, it'll be cathartic [18:15:38] Gone for tonight a-team - See you tomorrow ! [18:15:47] laters joal! [18:16:35] milimetric: jajaja no way [18:16:40] milimetric: that is a april's fool [18:16:45] :) it is [18:16:50] but it's so funny [18:16:59] and totally true in the figurative :) [18:17:50] nuria: oh, and guy fieri did get added to that repository in response: https://nodesource.com/blog/is-guy-fieri-in-your-node-js-packages/ [18:18:17] lol [18:18:18] so good [18:20:07] hahahahha milimetric i have only started this article and it is so good [18:23:54] :) [18:26:06] (03PS1) 10Nuria: Bump up pageviews.js to version that supports pagecounts [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/345197 (https://phabricator.wikimedia.org/T149358) [18:26:15] 10Analytics, 10Analytics-Cluster: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#3138061 (10Ottomata) @elukey When provisioning the main-* clusters, we decided to use RAID-10 instead of JBOD. This was certainly the right choice for the main... [18:26:23] breakdowns working on prototype bar now, take a look: https://analytics-prototype.wmflabs.org/#/contributing [18:26:35] hahaha [18:26:37] "I opened up babel-core in vim, then turned off my computer because Ctrl-C wasn’t exiting" [18:26:51] I took offense to that [18:26:57] 10Analytics, 10Fundraising-Backlog: Storage for banner history data - https://phabricator.wikimedia.org/T161635#3137911 (10DStrine) @Nuria I just added you to an email about the legal requirements. TLDR: we are allowed to keep the data for now. We need a place to park the data so we can aggregate it. That will... [18:27:02] I've opened 500MB files in vim and they're fine [18:28:03] ooh, fullscreen worked with that breakdown out of the box [18:28:27] I'm good at building something between a prototype and real code, meaning I'm useful to *nobody* hahahaha [18:29:37] milimetric: that guy fieri thing isn't true, is the rest of it? [18:29:38] haha [18:30:03] oh it was true! [18:32:42] no :) I'm not sure, you'll have to check, I think the point of the post is to get you to check [18:32:44] AndyRussG: still there? [18:33:00] ottomata: yep! [18:33:14] but the guy fieri thing is a joke, they added it with a commit message like "sorry for the bloat <>" [18:33:25] (the babel-core team) [18:33:37] next quarter we are going to start thikning and implementing https://phabricator.wikimedia.org/T159170 which probably starts with https://phabricator.wikimedia.org/T153328 [18:33:48] eventlogging data is already in hdfs [18:33:53] querying from hive/spark is possible [18:34:00] we delete by default everything older than 90 days [18:34:12] but as part of those tickets we'll have to be smarter with retention, like we are in MySQL now [18:34:15] so [18:34:34] if we had queryable EL tables in Hive that weren't necessarily deleted (for specific use cases like yours) [18:34:40] would that suffice? [18:34:48] or do you need special 'extracted' datasets [18:34:48] ? [18:39:32] ottomata: so, there are a lot of unknowns [18:39:50] And some knowns that I don't know :) [18:40:02] haha [18:40:09] ok, i'll just link to these from your ticket [18:40:18] o/ flying back to ny now [18:40:26] have a good flight! [18:40:36] milimetric: hi and bye :) [18:40:43] 10Analytics, 10Fundraising-Backlog: Storage for banner history data - https://phabricator.wikimedia.org/T161635#3138098 (10Ottomata) FYI, we are planning on improving Hive EventLogging integration next quarter: T153328 [18:41:00] What has been, and could continue to be extracted, from the eventlogging table, is the event content itself. So that gets rid of the metadata wrapper that's there in the hive tables [18:42:10] There was an e-mail discussion with legal about the contents of the banner history event itself. But since site access times have been fuzzed to prevent correlation with weblogs, and there's no information about articles browsed, or usernames, legal was OK with it being held longer than the 90 days [18:42:18] That doesn't make it ideal for indefinite retention though [18:43:11] Ideally it should also be aggregated [18:43:33] But decisions haven't been made about how to really do that, as far as I know [18:44:27] The knowns that i don't know is what queries have already been considered [18:44:59] That's 'cause the actual querying will be done by CPS (external analyst contractors helping w/ FR stuff) [18:45:30] Hopefully we can loop them in via Phab [18:48:09] But, in a nutshell, this would be an intermediate storage mechanism, that is more anonymized than what's in the Hive EventLogging tables, and does not automagically disappear after 90 days. By putting it in another space in hive, I think CPS will be able to try out some queries, find what's useful, and that'll be a first step towards knowing how to aggregate, and thus getting a stable, ongoing [18:48:12] mechanism to work with this data, and thereafter only store some aggregated form of it [18:49:47] The going beyond the 90-day bit is not because in general the data is needed in this intermediate form for longer than that, but rather because the decision of how to munge it is in flux, and we really should try stuff on data from the big year-end campaign (started at the end of Nov.) [18:49:55] Does that make any sense? ;p [18:55:49] 10Analytics, 10Analytics-EventLogging: Find an alternative query interface for eventlogging on analytics cluster that can replace MariaDB - https://phabricator.wikimedia.org/T159170#3138132 (10Nuria) [18:58:53] AndyRussG: ya makes sense [18:59:05] probably the step about extracting it to another space in hive will be easier after T153328 [18:59:05] T153328: Research Spike: Better support for Eventlogging data on hive - https://phabricator.wikimedia.org/T153328 [18:59:07] AndyRussG: I have responded via e-mail, i think there is a bit of homework to do here [18:59:22] then you can do your extraction + anonymization via hive queries [18:59:29] e.g. select ... from X insert into Y [18:59:37] all in hie [18:59:39] hive [19:07:42] nuria: thanks! I just responded also :) thanks for being thorough about this! Privacy is important :) [19:09:35] ottomata: thanks! interesting... [19:10:41] Ideally we should get CPS looped in here too... Apparently the should learn better IRC addiction skills... [19:29:54] AndyRussG: sharing data with external parties w/o security having it audited is really not possible [19:30:12] AndyRussG: let's talk about this in a phab ticket [19:30:35] nuria: OK... I mean, I'll just send the e-mail that I have almost ready... [20:08:15] 10Analytics, 10ChangeProp, 10EventBus, 06Services (done): Memory leak in Change-Prop - https://phabricator.wikimedia.org/T153215#3138302 (10Pchelolo) 05stalled>03Resolved After upgrading the `node-rdkafka` to a newer version and using a different approach to delivery report memory doesn't leak any more... [20:43:55] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3138367 (10Samwalton9) Good news is that some (not all) of my edits from the 15th showed up in the dat... [22:24:33] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3138670 (10Samwalton9) They've showed up now. Results: | Diff | Links added(+)/removed(-) this diff...