[00:21:00] (03PS1) 10Mayakpwiki: Add userName to ServerSideAccountCreation whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) [00:21:02] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [00:22:30] 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Product-Analytics: Identify next steps needed for Product Analytics to approve switching mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T238434 (10leila) [00:22:53] 10Analytics: Label high volume bot spikes in pageview data as automated traffic - https://phabricator.wikimedia.org/T238357 (10leila) [00:24:00] 10Analytics, 10Discovery, 10Operations, 10Recommendation-API: Run swift-object-expirer as part of the swift cluster - https://phabricator.wikimedia.org/T229584 (10leila) [00:24:25] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Performance-Team, and 2 others: Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10leila) [00:24:52] 10Analytics, 10Article-Recommendation: Make endpoint for top wikis by number of articles - https://phabricator.wikimedia.org/T220673 (10leila) [00:26:59] 10Analytics, 10Discovery, 10Operations, 10Article-Recommendation, 10Patch-For-Review: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10leila) [00:29:39] 10Analytics, 10Analytics-EventLogging, 10Research-Backlog: 20K events by a single user in the span of 20 mins - https://phabricator.wikimedia.org/T202539 (10leila) @Nuria do you expect this task to be addressed by the better bot detection approach you're implementing? If yes, I'd like to close it. [00:31:26] 10Analytics, 10Article-Recommendation, 10Patch-For-Review: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10leila) [00:31:42] 10Analytics, 10Operations, 10serviceops-radar, 10Article-Recommendation, and 3 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10leila) [00:45:06] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10Neil_P._Quinn_WMF) I too am requesting Kerberos credentials for the stat and notebook machines. My username is `neilpquinn-wmf`. [03:22:36] 10Analytics, 10Analytics-EventLogging, 10Research-Backlog: 20K events by a single user in the span of 20 mins - https://phabricator.wikimedia.org/T202539 (10Nuria) It will be initially deployed just for pageviews so not quite yet. [03:22:41] 10Analytics, 10Analytics-EventLogging, 10Research-Backlog: 20K events by a single user in the span of 20 mins - https://phabricator.wikimedia.org/T202539 (10Nuria) The heuristics would work however so we just need to think how would we plug it in this pipeline [03:23:27] (03CR) 10Nuria: [C: 03+2] Add userName to ServerSideAccountCreation whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [05:09:55] PROBLEM - Check the last execution of drop-mediawiki-siteinfo_namespaces-dumps on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit drop-mediawiki-siteinfo_namespaces-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:07:09] PROBLEM - Check the last execution of drop-mediawiki-pages_meta_history-dumps on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit drop-mediawiki-pages_meta_history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:47:57] (03PS1) 10Nuria: Setting logging level progrmatically in cassandra loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/551955 (https://phabricator.wikimedia.org/T236698) [06:51:53] (03CR) 10jerkins-bot: [V: 04-1] Setting logging level progrmatically in cassandra loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/551955 (https://phabricator.wikimedia.org/T236698) (owner: 10Nuria) [07:07:47] PROBLEM - Check the last execution of drop-mediawiki-pages_meta_current-dumps on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit drop-mediawiki-pages_meta_current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:09:03] good evening nuria! :) [07:09:17] elukey: wowow time tunnel [07:09:22] ahhahah yes [07:13:01] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Setup Config:Dashiki:WMCSEdits on meta wiki - https://phabricator.wikimedia.org/T236223 (10srishakatux) 05Open→03Resolved (thanks @Milimetric for the ping on this! ) [07:13:04] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Develop a tool or integrate feature in existing one to visualize WMCS edits data - https://phabricator.wikimedia.org/T226663 (10srishakatux) [07:15:02] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Setup a proxy "wmcs-edits" for a dashiki instance - https://phabricator.wikimedia.org/T237481 (10srishakatux) 05Open→03Resolved (there is nothing more left to do here) [07:15:06] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Develop a tool or integrate feature in existing one to visualize WMCS edits data - https://phabricator.wikimedia.org/T226663 (10srishakatux) [07:59:59] RECOVERY - Check the last execution of drop-mediawiki-siteinfo_namespaces-dumps on an-coord1001 is OK: OK: Status of the systemd unit drop-mediawiki-siteinfo_namespaces-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:00:17] RECOVERY - Check the last execution of drop-mediawiki-pages_meta_current-dumps on an-coord1001 is OK: OK: Status of the systemd unit drop-mediawiki-pages_meta_current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:00:54] goooood [08:01:26] we are missing one recovery [08:03:23] RECOVERY - Check the last execution of drop-mediawiki-pages_meta_history-dumps on an-coord1001 is OK: OK: Status of the systemd unit drop-mediawiki-pages_meta_history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:25:22] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10elukey) ` elukey@krb1001:~$ sudo manage_principals.py create conniecc1 --email_address=cchen@wikimedia.org Principal successfully created. Successfully sent email to cchen@wikimedia.org elukey@... [09:19:21] * elukey afk for a bit [10:08:41] (03CR) 10Milimetric: [C: 04-1] "This might be me, but I did try to run it on stat1004 as well as my local. Python environments will forever be a mystery to me, can you s" (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 (owner: 10Fdans) [10:14:54] milimetric: you have to run pip3 install python-dateutil before running the script :) [10:15:00] thanks for the review! [10:58:39] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 7 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10dr0ptp4kt) Hi team, I found one reviewer somewhat serendipitously. @pmiazga will provide a review of https:... [10:59:14] (03PS2) 10Fdans: Add granularity option to schedule monthly jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 [11:00:47] fdans: do you remember if karma is all messed up on dashiki? I'm just getting a huge printout of pageviews by country data and then it disconnects [11:01:31] milimetric: woah, I don't remember ever touching karma on dashiki, no idea [11:01:37] and fdans: I'm sure I'm being super stupid with the python script, but it's not the dateutil, I had that and the script doesn't throw errors, it just runs the tests and quits printing the usage [11:01:40] karma seems all messed up everywhere though [11:02:29] milimetric: this is the command I was using [11:02:32] bin/oozie-time-intervals -b 20191118 -s 20151009 -e 20190516 -u 1 -g monthly [11:02:32] heh, yeah, stupid thing [11:04:21] fdans: oh it's just a mismatch of param names in your example command, just update that: ./bin/oozie-day-intervals --backfill-start 20191104 --start 20151009 --end 20190727 -u 20 [11:04:41] cool, thanks, it works with the right param names, of course, sorry I didn't see that [11:05:13] fdans: but is there any way to prevent the tests from running every time? [11:05:40] milimetric: we decided to run them with every run ¯\_(ツ)_/¯ [11:05:51] oh ok, sounds fine [11:07:01] (03PS3) 10Fdans: Add granularity option to schedule monthly jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 [11:07:06] milimetric: just updated the example [11:11:45] (03PS2) 10Milimetric: Add grouped option to datasets api [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/551941 (https://phabricator.wikimedia.org/T236941) [11:15:03] (03CR) 10Milimetric: "Ready for review, to test just run a static server and browse to http://localhost:8000/src/layouts/metrics-by-project/#projects=rowiki,svw" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/551941 (https://phabricator.wikimedia.org/T236941) (owner: 10Milimetric) [11:22:47] (03PS4) 10Milimetric: Add granularity option to schedule monthly jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 (owner: 10Fdans) [11:22:50] fdans: sorry I was super confusing before, added another patch with the right example [11:22:56] fdans: can I merge? [11:23:04] milimetric: yes please! [11:23:07] thank you [11:23:56] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add granularity option to schedule monthly jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 (owner: 10Fdans) [11:24:34] (needs parents) [13:40:04] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Direct link generator to reports in Superset has the incorrect hostname - https://phabricator.wikimedia.org/T238461 (10elukey) @kzimmerman should work now, please check when you have a moment to confirm :) [13:40:33] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Direct link generator to reports in Superset has the incorrect hostname - https://phabricator.wikimedia.org/T238461 (10elukey) [13:51:58] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10jlinehan) New perspectives for the new day: #### Take 1 I could see a good argument for keeping a dichotomy, but... [13:56:33] * elukey errand for a bit! [13:57:21] (03PS1) 10Joal: Fix cassandra logging [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552067 (https://phabricator.wikimedia.org/T236698) [14:19:22] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 7 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) Our goal this quarter is to get the general idea approved so we can make it work with all the oth... [14:19:59] (03CR) 10Joal: [V: 03+2] "Tested on cluster with 1 instance of uniques-daily job, log size is decresed by 95%:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552067 (https://phabricator.wikimedia.org/T236698) (owner: 10Joal) [14:24:51] (03PS2) 10Joal: Fix cassandra logging [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552067 (https://phabricator.wikimedia.org/T236698) [14:40:20] Hey folks. I'm having trouble connecting to gerrit from stat1007. Could that be related to kerberos? [14:55:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) Hm, interesting. The main problem I see with Take 2 is the URIs. I want the relative schema URIs to st... [15:07:00] hey teammm [15:08:28] 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Operations, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [15:10:05] helloo [15:14:19] (03CR) 10Mforns: [C: 04-1] Add userName to ServerSideAccountCreation whitelist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [15:17:34] (03PS5) 10Fdans: Add granularity option to schedule monthly jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 [15:17:39] (03CR) 10Fdans: [V: 03+2] Add granularity option to schedule monthly jobs (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551841 (owner: 10Fdans) [15:26:22] (03CR) 10Mforns: [V: 03+2] Add the MobileWebUIActionsTracking schema to EventLogging whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/541946 (https://phabricator.wikimedia.org/T234563) (owner: 10MNeisler) [15:26:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10jlinehan) >>! In T206789#5678442, @Ottomata wrote: > That means all $refs and event $schema IDs need to be relativ... [15:38:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) Hm, you might be onto something with this permanent vs experimental idea. For example, should things li... [15:40:51] milimetric: o/ would you want to review https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/549652 (eventgate stream config stuff)? [15:40:58] if not petr has +1ed [15:41:00] up to you [15:43:59] looking [15:48:19] I see a huge number of event.mediawiki_api_request rows with meta.uri=null. Is this to be expected, or maybe a bug? [15:58:00] 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Operations, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10akosiaris) >>! In T236386#5672742, @Ottomata wrote: > @Joe @akosiaris @ema I'd like to move forward with these patches this... [15:58:04] elukey, netflow sanitization failed... :C I troubleshot and created a patch that will hopefully fix it: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552082 [15:58:11] elukey, can you have a look please? [15:59:32] 10Analytics, 10Analytics-Wikistats: Add multilanguage ability to Wikistats - https://phabricator.wikimedia.org/T238752 (10fdans) [15:59:37] 10Analytics, 10Analytics-Wikistats: Add multilanguage ability to Wikistats - https://phabricator.wikimedia.org/T238752 (10fdans) p:05Triage→03High [15:59:57] 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Operations, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) Thank you! I like your suggestions on the kafka producer TLS one, will implement. Joe can help with the rest tod... [16:00:07] hmm ApiMain::requestLog doesn't set `uri`, so I suppose this is intentional. Don't know what I'll be able to do with the events, in that case. [16:02:05] mforns: ah snap! I can merge but jenkins is still complaining [16:02:11] elukey, ok [16:09:17] ok, elukey, Jenkins stopped freaking out [16:14:03] thanks a lot elukey :D [16:16:28] awight, don't know if this is expected or not... [16:18:59] awight, but it looks like all rows are null, no? [16:19:17] can't you use the params field? [16:19:20] to what level are UAs NDA worthy? For example, is it okay if I make this https://phabricator.wikimedia.org/T199666 public? [16:19:41] mforns: puppet just ran, all deployed [16:19:54] elukey, thanks \o/ [16:20:10] now I have to re-run :[ [16:20:27] mforns: /o\ okay I somehow missed that, thank you! [16:21:27] np! [16:25:59] PROBLEM - Check the last execution of eventlogging_to_druid_netflow-sanitization_hourly on an-coord1001 is CRITICAL: NRPE: Command check_check_eventlogging_to_druid_netflow-sanitization_hourly_status not defined https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:27:42] addshore, hm I don't know... I'm not sure the UA of a bot that is selfidentifying qualifies as data that we collect from our users... but to be sure, I'd ask to legal? [16:27:46] ah ok some stuff got removed, this will clear when puppet runs on icinga --^ [16:28:09] addshore, I guess if instead of listing the full UA the task description was mentioning only the bot name, then it would be fine! [16:28:21] mforns: yes, I might do that in future :) [16:28:32] I'll leave those old ones marked with NDA for now [16:28:42] elukey, oh... I forgot to absent........ :[ [16:28:52] should I do it now? [16:30:18] addshore, cool [16:30:41] mforns: need to check but don't worry, should be ok [16:31:36] elukey, because by moving the params from job_config to root, I implicitly removed the hourly job without absenting first... didn't notice [16:32:22] theoretically the only one that left loose things was netflow-sanitization-hourly [16:34:43] mforns: let's check! [16:35:07] could be a good ops-week task to check the systemd timers statys [16:36:20] (03PS2) 10Mforns: Add userName to ServerSideAccountCreation whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [16:38:50] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Mayakpwiki, I changed the tab character to spaces, so I could proceed with the Analytics deployment train (Wednesdays) and include this ch" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [16:59:28] hi mforns - have you started train or not yet (refinery-sourcE) [16:59:40] joal, not yet! [16:59:45] \o/! [16:59:48] :] [17:01:36] I'd like nuria to review https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/552067/ before if possible [17:01:41] mforns: --^ [17:02:03] ok [17:07:52] 10Analytics, 10DBA: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10Ottomata) [17:07:54] 10Analytics-EventLogging, 10Analytics-Kanban: Sunset MySQL data store for eventlogging - https://phabricator.wikimedia.org/T159170 (10Ottomata) [17:08:00] 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Ottomata) [17:08:02] 10Analytics-EventLogging, 10Analytics-Kanban: Sunset MySQL data store for eventlogging - https://phabricator.wikimedia.org/T159170 (10Ottomata) [17:09:40] 10Analytics, 10DBA: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10Ottomata) T231858 is done so this is unblocked, woohoo! [17:09:52] 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Ottomata) T231858 is done so this is unblocked, woohoo! [17:15:42] 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Ottomata) @elukey can we move this to Analytics-Kanban? [17:20:07] 10Analytics, 10Analytics-Kanban: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) [17:20:49] 10Analytics, 10Cleanup, 10Event-Platform, 10Gerrit, and 4 others: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10hashar) [17:22:09] (03CR) 10Mayakpwiki: "> Mayakpwiki, I changed the tab character to spaces, so I could" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/551945 (https://phabricator.wikimedia.org/T238683) (owner: 10Mayakpwiki) [17:22:19] (03CR) 10Nuria: [C: 03+2] Fix cassandra logging [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552067 (https://phabricator.wikimedia.org/T236698) (owner: 10Joal) [17:23:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Logging level of cassandra should be warning or error but not debug - https://phabricator.wikimedia.org/T236698 (10Nuria) [17:51:53] * elukey off! [18:25:44] (03Abandoned) 10Nuria: Setting logging level progrmatically in cassandra loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/551955 (https://phabricator.wikimedia.org/T236698) (owner: 10Nuria) [18:37:43] 10Analytics, 10Repository-Admins, 10User-MarcoAurelio: Deletion of limn-flow-data repository - https://phabricator.wikimedia.org/T228981 (10MarcoAurelio) a:03MarcoAurelio Repository deletion has been proven a bit problematic recently so I'll just archive the repo for now. [18:39:05] joal, can you help me test the hdfs-cleaner if you're still working today? [18:39:24] mforns: in 20 mins, 1-1 now :) [18:39:31] thanks! [18:44:01] uou I got 205 bounce action notification emails from mailman in 10 minutes... [18:45:27] maybe they're doing some cleaning [18:57:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) Ok no branch, Dan convinced me, I think that will just confuse more people that 2 repos. We could just... [19:05:59] mforns: when you tested fdans changes for the scheduler , did you run teh code from: nuria@stat1007:/srv/deployment/analytics/refinery/bin$ [19:06:04] mforns: or somewhere lese? [19:06:09] *else? [19:06:42] nuria, I didn't run it, just left some comments that I thought needed change [19:06:51] didn't review after changes I think [19:12:37] nuria: being run from my home refinery [19:12:57] fdans: hola, ya, i saw that cause it hasn't been deployed yet [19:20:15] mforns: here I am! [19:20:22] mforns: cave? [19:20:27] joal, yea! [19:20:29] omnw [19:20:41] omnomonwomww [19:27:48] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10mepps) I have no doubts about this change generally, but Fundraising and Fundraising tech rely on this data heavily and will have just started the annual fundraiser on December 2nd. Co... [19:37:06] 10Analytics, 10Event-Platform, 10Operations, 10Wikimedia-Logstash, 10observability: Move eventgate logs to new logging infrastructure - https://phabricator.wikimedia.org/T225129 (10Ottomata) a:03Ottomata [19:39:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 8 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) [19:40:12] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Nuria) @mepps the data you need should be present in kafka even if hadoop has a several hour outage, which we hope does not happen. >We are also transitioning to consuming data from t... [19:43:14] 10Analytics: Move Eventstreams to kubernetes deployment pipeline - https://phabricator.wikimedia.org/T227122 (10Ottomata) [19:43:23] 10Analytics, 10Release Pipeline, 10Patch-For-Review, 10Release-Engineering-Team (Pipeline), 10Services (watching): Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) [19:45:49] 10Analytics, 10Operations, 10Wikimedia-Logstash, 10observability, and 3 others: Move eventstreams logging to new logging pipeline - https://phabricator.wikimedia.org/T219922 (10Ottomata) 05Open→03Declined We'll be moving EventStreams to k8s next quarter, which will take advantage of new logging pipelin... [19:47:29] 10Analytics, 10Analytics-EventLogging: db1107 and db1108 buffer pool size configuration - https://phabricator.wikimedia.org/T224291 (10Ottomata) 05Open→03Declined Since we'll be doing {T159170} soon, declining this. [19:50:05] 10Analytics, 10Operations, 10serviceops-radar, 10Article-Recommendation, and 3 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Ottomata) 05Stalled→03Resolved a:03Ottomata This is now supported via Kafka, Swift and an Oozie workflow. {T... [19:51:28] 10Analytics, 10Discovery, 10Operations, 10Article-Recommendation, 10Patch-For-Review: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Ottomata) 05Open→03Resolved a:03Ottomata This was finished back in July.... [19:51:32] 10Analytics, 10Operations, 10serviceops-radar, 10Article-Recommendation, and 3 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Ottomata) [19:54:47] 10Analytics: Refine Monitor should be a systemd timer such if process cannot start we get notified - https://phabricator.wikimedia.org/T210759 (10Ottomata) 05Open→03Resolved Hm? They are timers no? Perhaps this just changed in the last year. ` [@an-coord1001:/home/otto] $ systemctl list-timers | grep moni... [20:04:21] (03PS3) 10Srishakatux: Modify WMCS queries [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/551690 (https://phabricator.wikimedia.org/T232671) [20:08:27] Wow - Iran has blocked internet - [20:08:27] https://turnilo.wikimedia.org/#pageviews_hourly/4/N4IgbglgzgrghgGwgLzgFwgewHYgFwhLYCmAtAMYAWcATmiADQgYC2xyOx+IAomuQHoAqgBUAwoxAAzCAjTEaUfAG1QaAJ4AHLgVZcmNYlO4B9E3sl6ASnGwBzYkryqQUNLXoEATAAYAjACcpH5+pP4iPj54kdE+AHSRPgBaksTYACbcvoHBXmF+EVExkQmRKQC+ALrlDGpaOq5oNBD2kobGBOSYMNhN6pJw5Bg43C2SYIgwjiogAJI0tiCVTNiYnlKIUMRVTFCaSGhOLhra3BZM6RBs2FBYuARm5yB2C9gwCLQQGtwACn4AIpIoJg6PhQO1TOYrvpmPVuJdrrcRhcIIYhndu [20:08:33] HAoOQ0pdWjVCFcvvg3ggEDtXAoINNnOCjAo0jjuJBiAB3ExdHr0JhSEEsdBg2GnAjpIxwd7c8CTBogAknBpsLEwQyy5YgTQtEjpf7Qm53I5qjXYLUAZRBnnA1PZnN6kgQxAcGRJ7wQTEoEDslCQns8pPJQA [20:08:36] Arf sorry [20:08:50] https://gist.github.com/jobar/3e9659068ab3b751dfc42cfb28650e56 [20:08:52] better [20:16:34] 10Analytics, 10Operations, 10Traffic, 10User-Elukey: Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls - https://phabricator.wikimedia.org/T177927 (10Ottomata) 05Open→03Declined Old task, I think we aren't likely to do this. Declining, feel free to reopen i... [20:32:34] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10mepps) @Nuria Thanks for the quick and thorough response! To confirm, I read your comments as saying no kafka streams will either have downtime on December 2nd or require authenticatio... [20:33:12] neilpquinn: yt? [20:33:34] (03PS1) 10Mforns: Make hdfs-cleaner resilient to in-flight file deletion [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552128 (https://phabricator.wikimedia.org/T238304) [20:38:08] (03CR) 10Ottomata: Make hdfs-cleaner resilient to in-flight file deletion (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/552128 (https://phabricator.wikimedia.org/T238304) (owner: 10Mforns) [20:42:59] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Milimetric) I'm also liking the idea of experimental, and I think `experiment` is actually really nice and concise... [20:52:25] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) @milimetric how do you feel about 'exploratory' vs 'experimental'? While I like the idea that the non-r... [21:03:02] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Nuria) @mepps, Right, kafka is not affected [21:22:10] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Nuria) Some ideas: foundational/experiments -> my favorite foundational/analytical foundational/secondary core/e... [21:25:26] Well this is weird. For some reason I'm getting Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating get_geo_data(client_ip) https://www.irccloud.com/pastebin/ClKhrlR4/ [21:30:31] bearloga: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/refine_webrequest.hql#L52 [21:31:56] (03PS1) 10MarcoAurelio: whitelist: Add new ge.wikimedia.org [analytics/refinery] - 10https://gerrit.wikimedia.org/r/552138 (https://phabricator.wikimedia.org/T236389) [21:32:22] bearloga: ah, wait those two are same func , one sec [21:33:37] (03PS2) 10MarcoAurelio: whitelist: Add new ge.wikimedia.org [analytics/refinery] - 10https://gerrit.wikimedia.org/r/552138 (https://phabricator.wikimedia.org/T236389) [21:39:08] nuria: yeah it's the weirdest thing because I've never had a problem with that UDF before [21:54:09] bearloga: your issue is the exact one I describe here: https://phabricator.wikimedia.org/T238432 [21:54:47] bearloga: I thought it was related to MaxMind data not being present on an-coord1001, where HiveServer2 runs, and therefore preventing a loca-job to succeed [21:55:03] But, from Luca's comment on the task, it seems something else is at stake [21:55:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) @nuria I thought you didn't like 'experiments'? Doesn't that imply that the schemas there are intended... [21:55:20] I've not investigated more as of now, but it's on the list [21:56:10] bearloga: quick fix is to force a reduce, preventing the local job (and also being a lot slower and resource consuming) :( [21:56:33] Gone for real :) [21:59:17] nuria joal: I'm glad to know I'm not losing my mind. Hopefully this issue is not affecting webrequest refinement [22:41:26] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Nuria) >Doesn't that imply that the schemas there are intended to be for short term usage? mmm..this is hard but s... [23:00:25] ottomata: I am now! what's up? [23:08:21] PROBLEM - Check the last execution of hdfs-cleaner on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit hdfs-cleaner https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:59:53] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Add data quality metric: traffic variations per country - https://phabricator.wikimedia.org/T234484 (10Nuria) {F31115524} See event in iran as of today