[01:06:58] 10Analytics, 10Better Use Of Data, 10MediaWiki-API, 10Product-Analytics, 10Reading-Infrastructure-Team-Backlog: Load API request count and latency data from Hadoop to a dashboard - https://phabricator.wikimedia.org/T108414 (10kzimmerman) This is relevant to recent discussions about tracking content consu... [01:11:39] 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, 10EventBus, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10kzimmerman) [02:26:14] (03PS4) 10Milimetric: Use db_mapping to find the hostname [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) [02:26:16] (03PS2) 10Milimetric: Fix linting errors [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493285 [02:26:18] (03PS2) 10Milimetric: Add ipblocks_restrictions table to monthly sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) [02:28:15] (03CR) 10Milimetric: "Tested with --labsdb and without, found a few bugs, fixed. This should be ready to be merged, along with child changes for linting and ip" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric) [02:30:01] cool, I never did this with gerrit ^ and I think Christian showed me how to do it like 4 years ago :) (push a string of changes, then make more commits, rebase, squash, and push all changes at the same time) [04:07:10] (03CR) 10Nuria: "Let's decide if this is something we want to add to March sqooping." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) (owner: 10Milimetric) [06:13:28] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) [06:31:08] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: RFC: Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201963 (10Joe) >>! In T201963#4972858, @Ottomata wrote: > Can we close this? Yes, this ticket is just... [06:33:11] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10akosiaris) [06:33:19] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline - https://phabricator.wikimedia.org/T211247 (10akosiaris) 05Openβ†’03Resolved Resolving this, feel free to reopen [06:35:16] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Samwilson) We need #analytics to create a new beacon for us to use, and then we can add the required code to SVG Translate... [06:45:39] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) @Cmjohnson DHCP works fine and I can PXE boot, but then the Debian installer complains about "no partition found".. I checked via the... [06:54:07] hw raid seems misconfigured --^ [06:54:07] sigh [06:56:19] (03CR) 10Milimetric: "I think we can merge this along with the other changes in this chain. It's untested, maybe I could test it first." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) (owner: 10Milimetric) [06:56:40] * elukey sees a late milimetric [06:56:44] :) [06:56:51] elukey: I was just going to say good morning [06:56:55] I rarely get to do that :) [06:57:00] :D [06:57:10] ok, and now I'm gonna go crash hard into my pillow [06:57:15] * milimetric puts on his helmet [07:00:42] gnight! [07:40:18] I just stopped camus to drain the cluster [07:40:23] with systemctl stop camus-*.timer [07:40:25] \o/ [08:04:04] I am going to deploy the new Yarn config (but not restart the RM of course) [08:07:56] Good morning elukey [08:08:02] Will monitor cluster drain :0 [08:08:06] :) [08:08:33] bonjour! [08:08:46] config deployed, waiting for the best moment to failover etc.. [08:09:23] Still some automatic jobs (druid mostly, one search, and some users ones [08:09:29] yep [08:09:39] going to grab a coffee in the meantime :) [08:09:45] Cheers :) [08:20:40] joal: should we kill the pyspark jobs? [08:21:23] elukey: yes, the pyspark-shell jobs are from notebooks and should rerunnalbe easily [08:21:26] let me kill them [08:21:29] super [08:22:56] elukey: there still are 3 queries running, if you don't let's wait for the RU one to finish at least (it's big and is well advanced) [08:23:43] yep! [08:26:01] (03CR) 10Joal: [C: 03+2] "Looks good to me! Will test it this morning, +2 from a human-read perspective." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) (owner: 10Milimetric) [08:26:21] elukey: almost there I think :) [08:26:41] elukey: GO GO GO :) [08:27:15] gogogo [08:28:05] master is 1002 [08:28:21] k [08:29:44] done [08:29:52] Wow :) [08:29:55] FAST :) [08:30:21] elukey@an-master1001:~$ hdfs dfs -ls /user/yarn/rmstore [08:30:22] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 [08:30:22] Found 1 items [08:30:22] drwxr-xr-x - yarn hadoop 0 2019-02-28 08:29 /user/yarn/rmstore/FSRMStateRoot [08:30:32] \o/ ! [08:30:38] trying a spark-shell [08:31:20] everything looks good elukey :) [08:31:42] goooood [08:31:55] I'll wait a bit before starting the clean up of zookeeper [08:36:56] elukey: Do you mind if we try something before cleaning elukey? Like looking manually in ZK nodes to check that no new app data is there (safety first) [08:38:17] (03CR) 10Joal: [C: 03+2] "Looks good to me - Testing with ipblocks_restrictions." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric) [08:39:46] joal: sure sure [08:39:58] I am going to send an email to analytics@ explaining what we did [08:40:04] in case people notice something strange [08:41:21] ack! [08:51:27] ah wait I was wondering why jobs were not starting [08:51:35] I didn't re-enable camus :P [08:52:46] :) [08:54:56] joal: I have seen some jobs like https://yarn.wikimedia.org/cluster/app/application_1551342556468_0044 flowing [08:55:00] is it normal? [08:55:11] It is elukey :) [08:55:26] fiuuuuu [08:55:35] okok [08:55:39] it's me testing Dan's patch for sqoop + ipblocks-restrictions [08:55:43] I thought that yarn was getting crazy [08:55:46] Sorry, I should have mentionned [08:55:50] nono it is fine :D [09:14:02] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Hadoop Yarn stores a ton of znodes related to running/old applications - https://phabricator.wikimedia.org/T216952 (10elukey) Roll restart of RMs done. Sanity check: `application_1551342556468_0162` is currently running (camus-webrequest)... [09:14:06] joal: --^ [09:14:35] \o/ !!! Let's clean up ZK :) [09:14:48] Thanks a lot for sanity checking elukey :) [09:18:16] ah snap rmr in zookeeper cli doesn't support wildcars [09:18:20] wildcards [09:18:32] I wanted to avoid rmr to all the znodes in one go [09:19:09] elukey: we can go for a loop, but it might actually worse [09:19:34] in small batches should be fine, but it is a bit scary :D [09:19:42] I have already done it in the past though [09:19:52] right [09:39:29] joal: I am experimenting a simple bash loop of one rmr at the time + sleep 1 [09:39:52] it seems a nice pace, zk seems ok from what I can see in the metrics [09:39:58] it will take a lot of course [09:40:02] awesome [09:40:51] I think that a single rmr to the parent znode should work fine as well [09:41:02] elukey: I wouldn't mind :) [09:41:43] joal: do you think that I am currently too paranoid? [09:42:16] elukey: never too paranoid, but I wonder if it going for the loop wouldn't actually incur more load on zk than the global rmr [09:42:26] (over time of course, but maybe globally more?) [09:45:35] my main paranoia about the rmr to the parent znode is if zookeeper supports easily something that big without any issue [09:45:53] because with the current loop if I see anything weird I just control+c it [09:46:02] with the rmr command I am not sure [09:46:12] I hear you elukey [09:46:21] (like say zk crashes and leave its state inconsistent etc..) [09:46:22] I have no good advise here :) [09:48:05] !log restarting mediawiki-history-denormalize coordinator [09:48:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:56:02] !log restarting mediawiki-history-check_denormalize [09:56:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:57:45] !log restarting mediawiki-history-wikitext coordinator [09:57:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:59:10] joal: all 3 jobs restarted :) [10:00:49] Yay, thanks fdans :) [10:00:56] Moving the task on kanban [10:53:38] 10Analytics, 10Better Use Of Data, 10MediaWiki-API, 10Product-Analytics, 10Reading-Infrastructure-Team-Backlog: Load API request count and latency data from Hadoop to a dashboard - https://phabricator.wikimedia.org/T108414 (10Jhernandez) p:05Lowβ†’03Triage Thanks for clarifying @tgr @anomie. I've moved... [10:57:28] joal: looks nice! https://grafana.wikimedia.org/d/000000261/zookeeper?refresh=5m&panelId=42&fullscreen&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=main-eqiad&var-zookeeper_hosts=All&from=now-3h&to=now [10:58:15] Yeah! for once I can say: LEZZ (NODZ) :) [11:02:23] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Gilles) [11:02:40] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Gilles) a:05Gillesβ†’03None [11:27:10] (03PS5) 10Joal: Add change_tags to mediawiki_history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492320 [11:27:12] (03PS4) 10Joal: Update mediawiki-reconstruction with log info [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493012 [11:27:14] (03PS1) 10Joal: [WIP] Refactor mediawiki-page-history computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 [11:28:51] paused the zk cleanup, going afk for extended lunch! [11:28:53] ttl [11:33:03] heya teammm [11:33:33] Hi mforns :) [11:33:39] o/ [11:34:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Refactor mediawiki-page-history computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 (owner: 10Joal) [11:35:04] (03PS2) 10Joal: [WIP] Refactor mediawiki-page-history computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 [11:41:58] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Reading Depth, 10Readers-Web-Backlog (Tracking): Whitelist sample flags and page/rev ID fields for ReadingDepth schema - https://phabricator.wikimedia.org/T216096 (10mforns) @Tbayer thanks for the check! [11:42:26] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: [EventLogging Sanitization] Update EL sanitization white-list for field renames in EL schemas - https://phabricator.wikimedia.org/T209087 (10mforns) @chelsyx thanks for the check! [13:12:00] (03CR) 10Joal: [V: 03+2 C: 03+2] "Tested on cluster, +2 (parent to merge first)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) (owner: 10Milimetric) [13:12:07] (03CR) 10Joal: [V: 03+2 C: 03+2] Fix linting errors [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493285 (owner: 10Milimetric) [13:14:14] (03CR) 10Joal: [V: 04-1 C: 03+2] "One error at test, documented here not to still milimetric work :)" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric) [13:24:16] elukey: Heya - please ping me when back :) [13:42:59] joal: o/ [13:43:08] Hi :) [13:43:18] sorry I had to do a bit more stretching than expected, my knee is starting to recover but sloooowly :) [13:43:38] elukey: no worries :) You can even take 1h more and come after :) [13:44:06] Knee recovery is a good news :) [13:44:25] I hope you don't suffer too much :S [13:45:06] I have been doing basically mobility and stretching for the past 2/3 weeks [13:45:10] it was super painful [13:45:14] :( [13:45:28] but really worth it, I should do some yoga sooner or later [13:45:52] I really +1 the idea of doing yoga :) [13:46:55] anyway, sorry for the digression :) [13:46:57] did you need me? [13:47:10] ah yes puppet [13:47:11] :P [13:47:12] I do! [13:47:47] Not only puppet fro my patches, but I'd need /srv/mediawiki-config on an-coord1001 please [13:47:50] elukey: --^ [13:48:05] elukey: This is mandatory to be able to sqoop using the dbmapping [13:48:53] it should be there [13:48:54] joal: weird, I ran a full sqoop and it downloaded data and everything [13:49:06] also joal, one of your patches looks like https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/493331/ no? [13:49:17] both against cloud and prod replicas on shards with all tables (except cu_changes on cloud) [13:49:27] joal: so what broke with the . in the hostname? [13:49:50] milimetric: my test with sqoop worked for labs but not prod-repls [13:50:16] but in any case, I'll patch it real quick to strip the dot. That should be safe to do without changing the analytics-mysql wrapper because .strip is a no-op if it doesn't find the character [13:50:18] milimetric: because of the . at the end of the host I assumed [13:50:26] joal: what command did you use, it might be something else [13:50:42] I just want to double-check [13:51:15] oh!!!! oooooops, I think I forgot to export PYTHON on the screen [13:51:25] milimetric: https://gist.github.com/jobar/9e1aad2c46aef20126a3cad8e8b570fd [13:51:28] so I did some tests and then opened a screen and did more, and those were invalid!! [13:51:51] ok, I'll fix and redo the test [13:51:57] milimetric: when changing the util.py for stripping the dot, let's patch analytics-mysql, no? [13:52:15] milimetric: I'm positive that everything works when the . is removed (just tested) [13:52:24] GoranSM, hi! [13:52:29] And by the way milimetric, we have data for ipblocks_restrictions for 2019-01 [13:52:33] And your patch is valid [13:52:49] oh, cool, you ran it in prod [13:52:56] ok, I'll just send the patch then [13:53:11] super milimetric :) [13:53:34] elukey: indeed, my patches are covered by the one you sent ... I'm gonna abandon :) [13:54:33] And I confirm elukey that /srv/mediawiki-config is not here on an-coord1001 :) [13:56:11] super weird, lemme check [13:56:26] (03PS5) 10Milimetric: Use db_mapping to find the hostname [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) [13:56:29] (03PS3) 10Milimetric: Fix linting errors [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493285 [13:56:31] (03PS3) 10Milimetric: Add ipblocks_restrictions table to monthly sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) [13:57:33] (03CR) 10Milimetric: Use db_mapping to find the hostname (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric) [13:58:10] ok, joal, good call on checking that, I logged into an-coord yesterday and forgot to check it [13:58:21] all patches should be good now [13:58:34] milimetric: error on your patch: strip should be after str() [13:58:47] dope [13:59:20] joal: mistery, an-coord already have the puppet code for the repo [13:59:28] :/ [13:59:43] (03PS6) 10Milimetric: Use db_mapping to find the hostname [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) [13:59:45] (03PS4) 10Milimetric: Fix linting errors [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493285 [13:59:47] (03PS4) 10Milimetric: Add ipblocks_restrictions table to monthly sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) [13:59:52] * joal looks awkwardly at the gods of puppetries [14:00:18] good thing you're here jo, I'm like 1/4 awake :) [14:00:36] anyway, when you guys figure out this puppet madness and merge all these patches, I'll choo choo [14:00:59] joal: ah no! I am stupid [14:01:01] as always [14:01:04] milimetric: no bother - I'd like all that to be deployed today ;) [14:01:06] a-team: if there's anything else to merge for the late romanian-time choo choo, speak now or forever [14:01:21] milimetric, heh, not on my side :] [14:01:24] k [14:01:29] elukey, wanna delete stuff? [14:01:31] milimetric: Indeed :) [14:02:00] mforns: 10mins, is it ok? [14:02:01] milimetric: let me check the last sqoop, and then I'll merge all that sqoozive! [14:02:04] elukey, sure! [14:05:36] (03CR) 10Joal: [V: 03+2 C: 03+2] "Tested, merge for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/492209 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric) [14:06:33] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493285 (owner: 10Milimetric) [14:07:09] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493286 (https://phabricator.wikimedia.org/T209549) (owner: 10Milimetric) [14:08:54] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491838 (https://phabricator.wikimedia.org/T205940) (owner: 10Joal) [14:09:55] joal: I am pretty sure that you didn't look enough on an-coord1001, I can see the repo! :P [14:10:55] jokes aside, what about the other two code changes? Do they need a chu chu train deployment first? [14:10:58] :D [14:11:59] elukey: I abandonned them as they were bundled in milimetric's one [14:12:14] And yes, it needs the train first [14:12:25] ack then [14:12:48] joal: the "train" is too formal, please call it with the appropriate name! [14:12:55] it is a tribute to fdans [14:12:58] :D [14:13:25] Yessir - I'll not mistake chuchu with `the train` anymore [14:13:36] (03PS1) 10Joal: Fix change_tag and change_tag_def table creation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493413 [14:13:52] elukey joal excuse me the proper way is to use the emoji πŸš‚ [14:13:54] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy (bugfix)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493413 (owner: 10Joal) [14:14:45] fdans: You'll have to excuse me on that one - I smilley a lot, but I am not an emojintelligent person [14:15:44] milimetric: all good on my side for chuchu :) [14:17:20] ok, everyone [14:17:24] WOOOOOOOOOOOOOOOOO WOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO [14:17:27] e@@@@@@@@@@@@@@@ [14:17:27] @@@"""""""""" [14:17:27] @" ___ ___________ [14:17:27] II__[w] | [i] [z] | [14:17:27] {======|_|~~~~~~~~~| [14:17:35] \oO--000'"`-OO---OO-' [14:17:37] TRAIN'S LEAVIN THE STATION, GET ON BOARD OR GET YOUR ARMS CHOPPED OFF [14:17:47] that was a bit extreme... [14:18:22] xDDD [14:18:57] ahahahahahahah [14:19:11] mforns: did this make it out last week or should I deploy -source too? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/484708/ [14:21:34] it was merged monday, so I guess it needs to go... but it's In Progress on kanban, I'll wait until you tell me what to do mforns [14:22:42] milimetric, hey yes, this can be deployed, no problem, but if it is the only thing that is there in refinery-source, then no need at all to deploy, it is still depending on other changes [14:23:04] mforns: ok, it's the only thing, so it's not useful by itself then? [14:23:14] no no, you can skip [14:23:16] :] [14:23:16] k, just refinery then [14:23:20] cool [14:23:45] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10mforns) Ok, starting the deletion now. [14:27:13] ooh, cool, scap now knows to deploy refinery to more places. I'm gonna look at that change to understand it better [14:29:19] oh :) targets is just a list of servers, well that's super simple [14:30:53] milimetric: please check that the scap deploy repo is updated [14:31:06] the last time I added the notebooks etc.. [14:31:20] doesn't really matter but better to keep things consistent :) [14:32:20] elukey: yeah, that's what I was saying, I git pulled and got the new servers, and saw it was pushing to all of them [14:32:30] https://www.irccloud.com/pastebin/4Q3JUIgu/ [14:33:05] PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is CRITICAL: CRITICAL [14:33:08] (it's in the process to git pull on <>/scap [14:33:26] ah nice! [14:33:34] checking notebook1003 [14:33:59] do you guys have to manually intervene whenever Fuse breaks? because.. that's a lot [14:34:24] yeah in theory I have a task to test autofs etc.. [14:34:31] but sometimes it hangs in strange ways [14:34:38] so a umount/mount needs to happen [14:34:43] that thing is not really stable [14:35:09] the other main issue is people abusing of notebooks [14:35:21] I need to find time to work on cgroups for those hosts [14:35:34] because that is what causes process to get killed etc.. [14:35:43] (in this case I think that fuse was killed by the oom) [14:36:39] I wonder if the deploy had anything to do with it [14:37:00] it looked like it was hanging for a bit, and then when the Fuse alert came, it moved on to the next host [14:37:11] could be coincidence, but I don't believe in that [14:38:39] milimetric: fuse was keeping the host busy trying to read from HDFS, and when it died your deploy finally had ressources to move on? [14:39:41] maybe yeah [14:40:07] !log refinery deployed with new sqoop logic and updated history/load job [14:40:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:40:21] elukey: now if you would please merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/493331/ we'll be all set [14:40:27] obviously only if it's ok [14:40:38] milimetric: deployed onto hdfs as well ? [14:40:49] joal: yeah, that's when I do the ! log [14:40:55] k :) [14:41:59] milimetric: depending on how you feel, would you mind having a look at https://phabricator.wikimedia.org/T178587? I'd like your views on how we move forward please :) [14:43:23] milimetric: yep doing it [14:43:53] joal: I will look but after standup, feeling dizzy now [14:44:11] no prob :) Thanks! [14:53:25] milimetric: done! [14:53:30] all deployed on an-coord1001 [14:53:36] Cc: joal [14:54:18] great, hm... what’s the sound of a train being done? [14:54:33] Oooooow ooow? [14:57:52] fdans: --^ [14:59:48] milimetric: that's more like the sound of jimi hendrix being done [15:03:19] RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is OK: OK [15:04:41] \o/ ! Monitoring tomorrow :) Thanks train-man [15:07:29] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Gilles) [15:17:17] 10Analytics, 10WMDE-Analytics-Engineering: Pyspark2 fails to read.csv when run with spark2-submit - https://phabricator.wikimedia.org/T217156 (10Ottomata) Hm, strange indeed! I would really expect `--files` (comma separated) to work. I can reproduce your problem! It seems that spark is not actually creating... [15:17:44] milimetric: actually we also need to restart mediawiki-history-load ozie job [15:17:56] milimetric: I'll do it after standup [15:18:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline - https://phabricator.wikimedia.org/T211247 (10Ottomata) Yahoo thank you! [15:18:43] ah no, it’s ok joal I’ll do it [15:19:24] milimetric: I moved the tasks you depployed today in done (yours as well) [15:19:34] gone for kids now - see you at standup [15:21:51] (03PS1) 10Bearloga: Update whitelisting for Android-related schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) [15:24:53] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Gilles) p:05Triageβ†’03Normal [15:50:11] joal: final count of znodes after cleanup - 18.9k :D [15:52:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Hadoop Yarn stores a ton of znodes related to running/old applications - https://phabricator.wikimedia.org/T216952 (10elukey) Final znode count: 18.9k (we started from 43.5k) [15:54:58] 10Analytics, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) Proposal for removal: `registry brokers services etc consumers` @Ottomata what do you think? [15:55:27] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) [15:58:27] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10Ottomata) don't know about registry, services or etc, but /brokers and /consumers should be leftover from when we might have had a... [16:06:28] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) /etc is my fault when I've set up burrow the first time, and registry/services seems to be @joal's slider test (so safe to... [16:09:06] elukey, the script finished, it looks good partition-wise! It had an error in the wmdebanner regex, it should have been camelCased, because it is about paths, and they are camelCased, so corrected it and it's running again. [16:12:52] mforns: nice! [16:27:14] elukey, k, it finished, wanna review the log? [16:28:00] mforns: sure [16:28:14] (I am in bc) [16:28:33] (03PS3) 10Ladsgroup: Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493207 (https://phabricator.wikimedia.org/T213894) [16:29:16] ok omw [16:39:01] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) Got down to: ` [zk: localhost:2181(CONNECTED) 37] ls / [zookeeper, yarn-leader-election, hadoop-ha, hive_zookeeper_namesp... [16:39:07] 10Analytics, 10Analytics-Kanban, 10Operations, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) [16:39:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Hadoop Yarn stores a ton of znodes related to running/old applications - https://phabricator.wikimedia.org/T216952 (10elukey) [16:40:10] cleaned up all stale zookeeper nodes [16:40:13] now it really looks tidy [16:40:18] * elukey feels good [16:42:33] elukey, 9 minutes per table... [16:43:05] elukey, will take about 1 day to finish... :( [16:43:56] elukey, I will write the puppet patch, but let's not merge it until this run finishes (tomorrow hopefully), right? [16:44:38] or monday would be ok too [16:45:28] ack! [16:51:32] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10mforns) The deletion is finished now. The event database only contains now the last 90 days... [16:52:32] (03CR) 10Addshore: [C: 03+2] Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493207 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup) [16:52:41] (03Merged) 10jenkins-bot: Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493207 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup) [16:53:38] (03PS1) 10Addshore: Fix typo in three similar userprops scripts [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493446 [16:53:41] (03CR) 10Addshore: [V: 03+2 C: 03+2] Fix typo in three similar userprops scripts [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493446 (owner: 10Addshore) [16:53:49] (03PS1) 10Addshore: lib: Update and add PHPDoc tags to all lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493447 [16:53:51] (03Merged) 10jenkins-bot: Fix typo in three similar userprops scripts [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493446 (owner: 10Addshore) [16:53:53] (03CR) 10Addshore: [V: 03+2 C: 03+2] lib: Update and add PHPDoc tags to all lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493447 (owner: 10Addshore) [16:54:01] (03Merged) 10jenkins-bot: lib: Update and add PHPDoc tags to all lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493447 (owner: 10Addshore) [16:54:04] (03PS1) 10Addshore: Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493448 (https://phabricator.wikimedia.org/T213894) [16:54:10] (03CR) 10Addshore: [V: 03+2 C: 03+2] Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493448 (https://phabricator.wikimedia.org/T213894) (owner: 10Addshore) [16:54:18] (03Merged) 10jenkins-bot: Move reads to multi-source db setup [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493448 (https://phabricator.wikimedia.org/T213894) (owner: 10Addshore) [16:54:51] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10mforns) @GoranSMilovanovic I found 2 schemas that I wasn't aware of, that I believe belong t... [16:56:13] (03PS3) 10Addshore: Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493212 (https://phabricator.wikimedia.org/T216613) (owner: 10Thiemo Kreuz (WMDE)) [16:56:28] (03CR) 10Addshore: [V: 03+2 C: 03+2] Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493212 (https://phabricator.wikimedia.org/T216613) (owner: 10Thiemo Kreuz (WMDE)) [16:56:33] (03PS1) 10Addshore: Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493452 (https://phabricator.wikimedia.org/T216613) [16:56:37] (03Merged) 10jenkins-bot: Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493212 (https://phabricator.wikimedia.org/T216613) (owner: 10Thiemo Kreuz (WMDE)) [16:56:39] (03CR) 10Addshore: [V: 03+2 C: 03+2] Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493452 (https://phabricator.wikimedia.org/T216613) (owner: 10Addshore) [16:56:49] (03Merged) 10jenkins-bot: Add strict type hints to several scripts and lib classes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493452 (https://phabricator.wikimedia.org/T216613) (owner: 10Addshore) [16:56:50] sorry for the extreme spam [16:56:53] :D [17:00:58] 10Analytics, 10Analytics-Cluster, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10Halfak) [17:01:54] a-team i have better use of data right now will miss standup [17:02:14] ping milimetric [17:02:16] ping fdans [17:03:46] nuria: omw to talk! wish me luckers! [17:04:01] fdans: k, please send e-scrum [17:06:44] 10Analytics, 10Analytics-Cluster, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10EBernhardson) To be clear, we are talking about https://github.com/AbiWord/enchant ? [17:12:02] ottomata: i suppose random extra note, i understand what you mean about SRE wanting deps to come from the OS, but in this case we are talking about a dictionary and not executable code that could have security problems(afaik) [17:15:54] 10Analytics, 10Analytics-Cluster, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10Halfak) Yes. And the pyenchant python library. [17:16:25] !log restarted mediawiki/history/load job: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0131840-181112144035577-oozie-oozi-C/ [17:16:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:16:56] fdans you're gonna be awesome, have fun [17:17:37] thank youuu [17:20:50] 10Analytics: Partition event-data daily instead of hourly - https://phabricator.wikimedia.org/T217350 (10JAllemandou) [17:33:21] 10Analytics: Partition event-data daily instead of hourly - https://phabricator.wikimedia.org/T217350 (10Milimetric) p:05Triageβ†’03Low [17:34:34] 10Analytics: Partition event-data daily instead of hourly (for sanitized data) - https://phabricator.wikimedia.org/T217350 (10Milimetric) [17:40:46] 10Analytics: Some event data should not get sanitized - https://phabricator.wikimedia.org/T217271 (10Milimetric) [17:40:59] 10Analytics: Some event data should not get sanitized - https://phabricator.wikimedia.org/T217271 (10Milimetric) p:05Triageβ†’03Normal [17:41:35] 10Analytics: Some event data should not get sanitized - https://phabricator.wikimedia.org/T217271 (10Milimetric) [17:41:39] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Milimetric) [17:42:21] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering: Pyspark2 fails to read.csv when run with spark2-submit - https://phabricator.wikimedia.org/T217156 (10Milimetric) a:03Ottomata [17:42:26] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering: Pyspark2 fails to read.csv when run with spark2-submit - https://phabricator.wikimedia.org/T217156 (10Milimetric) p:05Triageβ†’03High [17:42:55] 10Analytics, 10Analytics-Kanban, 10Performance-Team: ServerTiming schema value for duration is 0 - https://phabricator.wikimedia.org/T217111 (10Milimetric) p:05Triageβ†’03Normal [17:43:16] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog: CentralNoticeImpression refined impressionEventSampleRate is int instead of double - https://phabricator.wikimedia.org/T217109 (10Milimetric) p:05Triageβ†’03Normal [17:43:20] 10Analytics, 10Fundraising-Backlog: CentralNoticeImpression refined impressionEventSampleRate is int instead of double - https://phabricator.wikimedia.org/T217109 (10Milimetric) [17:43:27] 10Analytics, 10Performance-Team: ServerTiming schema value for duration is 0 - https://phabricator.wikimedia.org/T217111 (10Milimetric) [17:43:54] 10Analytics, 10Performance-Team: ServerTiming schema value for duration is 0 - https://phabricator.wikimedia.org/T217111 (10Milimetric) a:03Krinkle [17:45:10] 10Analytics: Spike [2019-2020 work] Airflow Study - https://phabricator.wikimedia.org/T217059 (10Milimetric) [17:46:38] 10Analytics: Spike [2019-2020 work] Airflow Study - https://phabricator.wikimedia.org/T217059 (10Milimetric) p:05Triageβ†’03Normal [17:47:03] 10Analytics: decouple analytics zookeeper cluster from kafka zookeeper cluster - https://phabricator.wikimedia.org/T217057 (10Milimetric) p:05Triageβ†’03Normal [17:47:44] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) FYI I solved the issue disabling the SD card support in: System Configuration -> Bios/Platform configuration -> System Options -> Usb... [17:47:46] 10Analytics: decouple analytics zookeeper cluster from kafka zookeeper cluster [2019-2020] - https://phabricator.wikimedia.org/T217057 (10Milimetric) [17:51:27] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Milimetric) p:05Normalβ†’03High [17:51:35] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Milimetric) p:05Highβ†’03Normal [17:53:45] mforns: just had a thought about something (let's discuss after meeting): how does druid ingestion handle map types? [17:53:58] do the keys need to be explicitly added as dimensions in the spec? [17:54:04] ottomata, it doesn't [17:54:23] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Milimetric) Not rude at all :) Piwiki is normally for independent websites deployed on labs (we have some exceptions). B... [17:54:30] it's a todo: https://phabricator.wikimedia.org/T208589 [17:54:34] ottomata, ^ [17:55:21] ok interesting. [17:55:41] in either case the desired map keys dimensions need to be explicitly specified then [17:55:42] hm [17:57:47] yes, they would... [18:07:21] 10Analytics, 10Operations, 10Services: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10Ottomata) [18:19:10] 10Analytics, 10Analytics-Kanban, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10Nuria) p:05Highβ†’03Triage [18:21:18] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Aklapper) >>! In T215478#4991782, @Milimetric wrote: > But where is this tool? How does it work? Also wondering... https... [18:28:57] * elukey off! [18:39:01] 10Analytics: Spike. GPU enabled computations. How to do that best - https://phabricator.wikimedia.org/T217367 (10Nuria) [18:39:16] 10Analytics: Spike {2019-2020]. GPU enabled computations. How to do that best - https://phabricator.wikimedia.org/T217367 (10Nuria) [18:39:33] 10Analytics: Spike [2019-2020]. GPU enabled computations. How to do that best - https://phabricator.wikimedia.org/T217367 (10Nuria) [19:29:56] (03PS5) 10Ottomata: Event(Logging) schema loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) [19:31:36] (03CR) 10Ottomata: "Nuria, I got rid of the singletons altogether. Since we moved to class based instead of static methods, things like schema baseURI and sc" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [19:47:47] (03PS6) 10Ottomata: Event(Logging) schema loader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) [20:39:08] joal: i don't suppose you are still around? [21:05:08] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10mobrovac) [21:06:16] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201068 (10mobrovac) [21:06:19] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: RFC: Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201963 (10mobrovac) 05Openβ†’03Resolved [21:06:56] ottomata: ok, looked at json changes, the way calsses are now the cache instance is not shared among loaders, emaning that each loader has its own instance of the cache [21:07:22] ottomata: and the objectmapper [21:08:31] ottomata: sorry, retyping: the way classes are now the cache instance is not shared among loaders, meaning that each loader has its own instance of the cache [21:08:49] ottomata: teh factory took care of making loaders shared cache instance and object mapper instance [21:10:02] yes [21:10:41] we could wrap with a factory for our specific production instance use [21:10:55] but, in general, we'll only instantiate one of those [21:10:57] per job [21:11:06] each job is configured for the type of data it refines [21:12:18] (03CR) 10Nuria: Event(Logging) schema loader (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492399 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [21:13:04] ottomata: ya, "practically" it would not matter, design wise i think enforcing 1 instance of cache and object mapper per application is more correct [21:14:11] aye nuria, which was your fetcher. [21:14:25] i think I don't mind the fetcher idea, i just didn't like the factory [21:15:05] i don't see how it helps not load static things at classloadtime [21:15:08] in a thread safe wauy [21:15:14] there's always some class loading that happens [21:15:25] we could have a Fetcher class [21:15:27] that has what you had [21:15:31] and has a static instance [21:15:37] ottomata: the factory ensures teh cache and mapper are shared among instances though, that its its main goal [21:15:40] ok [21:15:49] that's fine, thought it was trying to get rid of class loading static stuff [21:15:56] ottomata: also it controls the lifecycle yeah cause it does the instantiation [21:16:01] but we can do that without factory [21:16:02] no? [21:16:07] just Fetcher.getInstance [21:16:08] and [21:16:15] static instance = new Fetcher in class body [21:16:35] ottomata: to have 1 cache shared among two loaders? [21:18:35] ottomata: the easiest is to pass the cache as a dependency on instantiation new XYZloader("cache") and for that you need a class that controls instantiation and the creation of the cache for all loaders right? [21:18:44] 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Niharika) >>! In T215478#4991895, @Aklapper wrote: >>>! In T215478#4991782, @Milimetric wrote: >> But where is this tool?... [21:19:07] hang on, maybe have eventbus errors in prod... [21:32:11] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering: Pyspark2 fails to read.csv when run with spark2-submit - https://phabricator.wikimedia.org/T217156 (10GoranSMilovanovic) @Ottomata To copy to HDFS - you mean like in the [[ https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark#Spark... [21:45:38] 10Analytics, 10Growth-Team: Blank the event.page_title column in the editorjourney table in the Data Lake - https://phabricator.wikimedia.org/T217377 (10nettrom_WMF) [21:51:22] 10Analytics, 10Analytics-Cluster, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10EBernhardson) So, the difficulty here is going to be that this isn't just dictionaries that feed into python deps, there... [22:10:01] 10Analytics, 10Discovery, 10EventBus, 10Services, 10WMF-JobQueue: EventBus mediawiki outage 2019-02-28 - https://phabricator.wikimedia.org/T217385 (10Ottomata) [22:11:19] 10Analytics, 10Discovery, 10EventBus, 10Services, 10WMF-JobQueue: EventBus mediawiki outage 2019-02-28 - https://phabricator.wikimedia.org/T217385 (10Ottomata) [22:17:36] 10Analytics, 10Discovery, 10EventBus, 10Services, 10WMF-JobQueue: EventBus mediawiki outage 2019-02-28 - https://phabricator.wikimedia.org/T217385 (10hashar) [22:18:52] ottomata: yt? (question, no more EL talk) [22:19:00] ya [22:20:39] 10Analytics, 10Growth-Team: Blank the event.page_title column in the editorjourney table in the Data Lake - https://phabricator.wikimedia.org/T217377 (10Nuria) @nettrom_WMF data in hive cannot be modified without it being deleted and rewritten (that is right, no alters in hadoop, you can change a column type... [22:23:05] 10Analytics, 10Analytics-Cluster, 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10Halfak) My sense is that we can have a pretty good guarantee that the dictionaries are the dictionaries, but you're righ... [22:23:36] ottomata: ah sorry, had not seen https://phabricator.wikimedia.org/T217385 [22:29:19] PROBLEM - eventbus grafana alert on icinga2001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. [22:44:24] 10Analytics, 10Discovery, 10EventBus, 10Services, 10WMF-JobQueue: EventBus mediawiki outage 2019-02-28 - https://phabricator.wikimedia.org/T217385 (10Ottomata) Not all `events` fields from had their `meta` objects in logstash. Not sure why. I had to filter those out: Replaying: ` lang=bash grep '"meta"... [22:46:45] RECOVERY - eventbus grafana alert on icinga2001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. [22:50:23] 10Analytics, 10Growth-Team: Blank the event.page_title column in the editorjourney table in the Data Lake - https://phabricator.wikimedia.org/T217377 (10Nuria) Editorjourney data is not in the whitelist or am I totally spacing out? https://github.com/wikimedia/analytics-refinery/blob/master/static_data/eventlo... [23:03:42] 10Analytics, 10Growth-Team: Blank the event.page_title column in the editorjourney table in the Data Lake - https://phabricator.wikimedia.org/T217377 (10Nuria) Ok, @MMiller_WMF confirmed no entry is needed in whitelist. [23:07:53] 10Analytics, 10Better Use Of Data, 10Product-Analytics: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10kzimmerman) [23:33:11] 10Analytics, 10Growth-Team: Blank the event.page_title column in the editorjourney table in the Data Lake - https://phabricator.wikimedia.org/T217377 (10nettrom_WMF) 05Openβ†’03Declined We're fine with letting this data get purged as it otherwise would, so I'm closing this.