[06:49:02] cchen_: hi! When you are online ping me so we can go through your issue. I don't currently see any trace of your username on notebook1003 or notebook1003, are you following https://wikitech.wikimedia.org/wiki/SWAP#Access_and_infrastructure ? [06:54:43] * elukey runs errand for an hour, bbiab! [08:26:01] (03PS19) 10Fdans: Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:26:27] (03CR) 10Fdans: Add mediarequests hourly oozie job (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [10:12:35] first version of the hadoop workers reboot cookbook ready: https://gerrit.wikimedia.org/r/#/c/operations/cookbooks/+/530096/ [10:12:38] \o/ [10:13:08] in theory with that I will be able to just start the reboot procedure and then do other stuff [10:16:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Create a spicerack recipe to reboot the hadoop worker nodes - https://phabricator.wikimedia.org/T225297 (10elukey) [10:17:38] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Delayed jobs fail validation in eventgate - https://phabricator.wikimedia.org/T230049 (10elukey) [10:17:58] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Core Platform Team Workboards (Clinic Duty Team): Refactor EventBus mediawiki configuration - https://phabricator.wikimedia.org/T229863 (10elukey) [10:56:54] * elukey lunch! [12:29:00] mforns: o/ [12:29:23] I tested what Dan asked yesterday during standup for the wikistat caching, is it ok if I merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/529795/ ? [12:36:24] (ok I'll be bold and merge :) [12:47:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Tune Wikistats 2 Varnish caching - https://phabricator.wikimedia.org/T230136 (10elukey) ` :~ curl -I https://stats.wikimedia.org/v2 -s | grep cache-control :~ curl -I https://stats.wikimedia.org/v2/ -s | grep cache-control cache-control: max-age=10 :~ c... [12:48:02] looks good! [12:48:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Tune Wikistats 2 Varnish caching - https://phabricator.wikimedia.org/T230136 (10elukey) [12:48:59] elukey, sorry missed your ping [12:49:36] np! I added some data to the task [12:49:44] the caching behavior looks good to me [12:52:37] cool! [13:28:13] mforns: do you have by any chance the indexation json file that you used to test netflow data the first time? [13:28:25] hmmm... [13:28:27] I'd love to add to druid a day of data with the new fields [13:28:33] as separate datasource [13:28:39] so we can test it [13:30:29] elukey, I don't remember having created a json spec for netflow [13:30:48] I think I used HiveToDruid, no? Lemme check [13:30:49] mforns: ah did you run manually the spark job? [13:31:01] makes sense I am totally ignorant about it :) [13:31:02] I think so, but let me confirm [13:33:36] elukey, that's what I used, I think by copying-modifying-pasting you can do what you want: https://pastebin.com/xawEzYLu [13:36:18] I think you only need to change --dimensions and --metrics, right? Maybe --num_shards=2 if the size of the data has increased a lot? [13:37:36] Oh, hm, elukey, the job automatically names the druid datasource as: database_table, so if you want to ingest with a different name... [13:38:22] dunno... O.o [13:38:50] ah snap [13:38:56] I may override what already there [13:38:58] we should add the ability to pass a custom datasource_name to HiveToDruid [13:40:21] elukey, maybe the fastest way for now, would be to modify the code and hardcode your datasource name in HiveToDruid on your local machine, then scp HiveToDruid to i.e. stats1007, [13:40:57] and compile there, and change the jar path to your local jar path in stat1007 [13:41:15] ah yes [13:41:58] like in line 188: dataSource = s"${config.database}_${config.table}", add a suffix or sth [13:42:52] that could be the default if nothing is passed to HiveToDruid [13:52:44] mforns: since I am a bit ignorant, mvn build is sufficient to build a local refienery jar right? [13:53:24] no sorry it is not 'build' [13:54:43] mvn package [14:03:32] elukey, yes I think so! [14:16:29] it is interesting that I get [14:16:30] TestEventLoggingToDruid: [14:16:30] *** RUN ABORTED *** [14:16:30] java.lang.AbstractMethodError: [14:16:43] but before making any changes [14:19:49] elukey, wow [14:20:07] I just compiled refinery-source and all was fine [14:20:10] in stat1007 [14:20:51] I am retrying with mvn clean + package [14:21:04] maybe it was my local repo messed up [14:21:08] ok [14:21:13] yeah all good now :) [14:21:16] mforns: milimetric heeey should we merge this business? [14:21:16] https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/528134/ [14:21:35] ok, I never understood the difference between mvn package and mvn clean package [14:22:11] ah I did those separate [14:22:16] there is also the combo :D [14:24:53] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [14:25:22] thank youuuu mforns [14:25:30] no problemmmo [14:40:46] sorry having a hard time with kids today, working on and off as I can [14:41:16] The create table looked good fdans, I left comments on the oozie job, ping me if I’m slowing you down there [14:42:02] milimetric: yes i responded to your cr right? [14:44:03] just checked and we could upgrade turnilo from 1.8.1 to 1.17.0 :D [14:48:29] mforns: https://turnilo.wikimedia.org/#test_elukey_wmf_netflow \o/ [14:53:04] one day of data seems ~200MB on the historicals [14:54:32] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10elukey) Indexed one day (Aug 1st) of data in https://turnilo.wikimedia.org/#test_elukey_wmf_netflow with the new dimensions. The size looks good, will do other checks with Marc... [14:59:41] elukey, netflow in turnilo looking good! I get one error though when trying to split by Ip Src... [14:59:54] what error? [15:00:23] it works for me.. [15:03:13] mforns: --^ [15:04:40] elukey, it doesn't fail now! [15:04:53] it was sth related to the cardinality of the IP field [15:06:06] weird [15:06:59] mforns: I can try to index a week now to see what changes [15:07:15] ok [15:30:34] one week of data indexed [15:30:41] still looking good [15:50:18] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) >>! In T225005#5406176, @herron wrote: >>>! In T225005#5... [15:52:44] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10elukey) Andrew is on holidays, but it looks good to me! [15:55:31] cool then! [15:56:12] Hi analytics people :). I Just wondering if I'm understanding things right. If I built some query on turnilo on the webrequest_sampled_128 cube am I right in thinking that multiplying my hits by 128 gives me the approx number of actual hits? Or did I need to do something else special [15:57:56] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) >>! In T225005#5414274, @elukey wrote: > Andrew is on ho... [16:07:14] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10ayounsi) Looks good to me! Some data is missing but it seems to be an issue on the exporter side. [16:16:19] groceryheist: hello :) [16:16:30] elukey: hello [16:16:30] I can see a huge spark job in the queue, taking a ton of memory [16:16:38] o/ [16:16:54] there was lots of free memory :) [16:17:08] should I restart with less? [16:17:13] ah yes but we have regular jobs and other people might need it :) [16:17:15] it should finish soon [16:17:21] ah ok no then, let's wait [16:17:26] how many workers did you use? [16:17:29] 400 [16:17:57] ahahahah [16:18:04] please don't do it, max 128 [16:18:19] or a bit more, but with a lot of care [16:18:22] ak [16:18:25] even if we have spare resources [16:18:27] it just failed [16:21:09] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) >>! In T225005#5414291, @herron wrote: >>>! In T225005#5... [16:26:13] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10elukey) Completely ignorant about it, I'd loop in @jijiki :) [16:31:04] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10Pchelolo) @herron I believe this is the documentation for it htt... [16:31:36] * elukey off! [16:57:51] I feel like not enough bad things are said about Hue [17:09:41] mforns: any chance you're there to bc for a couple mins? [17:09:58] fdans, sure! [17:09:59] omw [17:41:33] (03PS20) 10Fdans: Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [17:48:10] (03PS21) 10Fdans: Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [17:52:15] (03PS22) 10Fdans: Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [17:58:46] !log backfilling mediarequests from 2019-5-16 to 2019-8-14 [17:58:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:15:26] (03CR) 10Milimetric: Add mediarequests hourly oozie job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [19:41:41] milimetric: already backfilled all the way from may, the folders look good [19:42:11] HOUR, YEAR, etc are coordinator variables, so it's no problem using them there [19:42:54] (I spent half my day yesterday deciphering oozie's docs) [20:46:19] 10Analytics, 10Analytics-Wikistats: Wikistats Bug: Pageviews for nl.wiktionary - https://phabricator.wikimedia.org/T230512 (10MarcoSwart) [20:56:10] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add mediarequests hourly oozie job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [21:08:43] 10Analytics: Wikistats: month on dashboard changes as we change projects - https://phabricator.wikimedia.org/T230514 (10Milimetric) [21:16:24] 10Analytics, 10Analytics-Wikistats: Wikistats Bug: Pageviews for nl.wiktionary - https://phabricator.wikimedia.org/T230512 (10Milimetric) The numbers make sense to me, did you split by agent type? You can see a lot of spider (crawler) traffic: https://stats.wikimedia.org/v2/#/nl.wiktionary.org/reading/total-p... [21:18:37] 10Analytics: Wikistats: month on dashboard changes on any redraw - https://phabricator.wikimedia.org/T230514 (10Milimetric) [21:38:33] have kafka-main firewalls been changed perhaps? I'm trying to `kafkacat -b kafka-main1001 -L` from stat1007 but all i get are timeouts [21:39:38] or i suppose better question, from which hosts am i allowed to look at the job queue topics? [21:45:15] oh silly me, kafka-main is the old dead cluster... [21:45:16] 10Analytics, 10Analytics-Wikistats: Wikistats Bug: Pageviews for nl.wiktionary - https://phabricator.wikimedia.org/T230512 (10MarcoSwart) I hadn't noticed that the default presentation has suddenly been changed from User to "User+Spider".