[04:37:41] 10Analytics, 10Analytics-Kanban, 10ExternalGuidance, 10Product-Analytics, 10Patch-For-Review: [Bug] `init` and `mtinfo` event counts drop drastically since June 17 2019 - https://phabricator.wikimedia.org/T227150 (10chelsyx) Thanks so much @Nuria! [05:57:16] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10elukey) For some reason the disk doesn't show as failed by megacli but: ` elukey@analytics1072:~$ ls /var/lib/hadoop/data/b ls: reading directory '/var/lib/hadoop/data/b': Input/output error ` [07:00:33] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10jcrespo) Chris, you will need to coordinate with @elukey principally, as he is the person in touch directly with users affected to... [07:03:16] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) We can do it anytime with 10/15 mins of heads up Chris (I need to stop replication and traffic to db1107 before you can op... [07:19:40] (03CR) 10Awight: [C: 03+1] "Implied CR+2 :-)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730) (owner: 10Nuria) [08:09:32] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10elukey) The hdfs-audit log error comes from this bit in /usr/lib/hive/bin/hive: ` # Make sure we're using a compatible version of Hadoop if [ "x$HA... [08:14:58] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10elukey) No users were logged so I have also done `apt-get purge hive; puppet agent -tv`, to get a clean version of hive. Issue persists. [08:44:40] fdans: o/ [08:44:53] hellooo [08:45:01] hola :) [08:45:16] can I ask you a favor? could you log into stat1004 and execute 'beeline' ? [08:45:31] I am interested to know if you get a logging error for hdfs-audit.log [08:45:45] something like [08:45:45] log4j:ERROR setFile(null,true) call failed. [08:45:45] java.io.FileNotFoundException: /usr/lib/hadoop/logs/hdfs-audit.log (No such file or directory) [08:47:52] fdans: --^ [08:48:04] elukey: omw! [08:49:00] elukey: everything looks good: [08:49:04] https://www.irccloud.com/pastebin/fSLg2Td4/ [08:53:37] fdans: even with --verbose? [08:53:39] oh my [08:53:53] (you are on stat1004 right?) [08:54:27] let me see [08:54:32] yes stat1004 [08:55:06] elukey: yeah verbose doesn't produce any errors either [08:55:30] really crazy, if I sudo as your user and try I get the error [08:56:18] thanks a lot for the test, I am a bit confused now [08:56:32] elukey: good luck! [08:57:55] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10elukey) Ok turns out this is only a problem that I have, Fran seems to get no errors.. Now I am really confused :D [09:39:57] (03PS1) 10Fdans: [wip] Add file extension and media classification to mediacounts job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522390 (https://phabricator.wikimedia.org/T225911) [09:42:38] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10elukey) [09:42:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey) [09:48:15] 10Analytics, 10Operations, 10Traffic: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10ema) [09:48:25] 10Analytics, 10Operations, 10Traffic: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10ema) p:05Triage→03Normal [09:52:08] (03PS1) 10Elukey: oozie/projectview/hourly: move to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) [09:52:30] 10Analytics, 10Operations, 10Traffic, 10User-Elukey: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) [10:19:21] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10elukey) >>! In T136858#5322320, @Milimetric wrote: > * the prompt doesn't show the database you're on, which can cause accidents. According to http... [10:39:09] * elukey lunch! [10:50:51] hey yall [12:07:44] 10Analytics, 10User-Elukey: Show IPs matching a list of IP subnets in Webrequest data - https://phabricator.wikimedia.org/T220639 (10faidon) p:05Normal→03Low >>! In T220639#5296561, @elukey wrote: >> How do we run this with a venv so that we can include Pytricia? > > Ideally if we had a deb package for th... [12:53:28] (03CR) 10Ottomata: [C: 03+1] oozie/projectview/hourly: move to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:20:11] (03CR) 10Fdans: [C: 03+1] oozie/projectview/hourly: move to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:26:53] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Beeline does not print full stack traces when a query fails - https://phabricator.wikimedia.org/T136858 (10Milimetric) > This bit seems not easily solvable, since the new option has been introduced in 2.3.0 (https://issues.apache.org/jira/browse/HIVE-1412... [13:30:11] (03CR) 10Milimetric: [C: 03+1] "I can test this if you like" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:35:32] (03CR) 10Elukey: "Thanks! I am about to launch a coordinator using /user/elukey for refinery and target dir, should be enough as test, I'll ask for help if " (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:36:58] (03PS5) 10Fdans: Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) [13:39:29] (03CR) 10Fdans: Add file extension and media type classification to media files UDF (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [13:39:32] (03CR) 10jerkins-bot: [V: 04-1] Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [13:40:11] milimetric: o/ I have a question about testing the projectview coordinator if you have time [13:40:17] (03CR) 10Milimetric: [C: 03+1] oozie/projectview/hourly: move to hive2 actions (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:40:47] elukey: course [13:40:52] (03CR) 10Elukey: ">" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [13:40:54] i knew intellij idea would halfass that refactor, hence the -1 [13:42:29] milimetric: I uploaded my version of refinery in /user/elukey, and I modified the coordinator.properties to write the legacy format under /user/elukey/archive.. but I noticed that the coord also writes in wmf.projectview_hourly, so I guess I'll have to create a similar table in elukey.projectview_hourly? [13:43:08] yeah, you have to override all that stuff, lemme take a quick look [13:43:53] when I test one of these jobs and I find things I need to override, I usually update the example usage: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/projectview/hourly/coordinator.properties#L4 [13:43:56] the quick way could be to upload the new coordinator.xml to hdfs (in the "official" refinery) and then relaunch the job :P [13:44:30] nono :) [13:44:34] ah yes but "destination_table" needs to be created somewhere [13:44:43] like in the elukey databse [13:44:49] otherwise the insertion will fail [13:44:58] or not? [13:46:07] so for this I'd override oozie_directory, start_time/stop_time, projectview_data_directory, destination_table, and archive_directory [13:46:39] yep, same thing that I thought [13:46:44] k, cool [13:46:53] and yeah, you need to create the table in your db [13:47:04] the table create script is in the hive folder, you know where? [13:47:43] we keep the target db out of these so you can run it with --database elukey [13:47:44] https://github.com/wikimedia/analytics-refinery/blob/master/hive/projectview/hourly/create_projectview_hourly_table.hql#L15 [13:48:16] (remember to change the location it's pointing to though) [13:48:25] https://github.com/wikimedia/analytics-refinery/blob/master/hive/projectview/hourly/create_projectview_hourly_table.hql#L36 [13:49:00] ah nice! [13:49:44] so something like hive -f create_projectview_hourly_table.hql --database elukey [13:50:36] ah also LOCATION [13:51:49] going to use /user/elukey/projectview/hourly [13:52:44] all good [13:54:00] milimetric: about the overrides: did you mean refinery dir rather than oozie? [13:54:47] for testing I usually leave the refinery dir as current, and change just the oozie dir to point to my deployed version, that way I can put -f only the oozie folder [13:55:03] ahhh [13:55:04] this obviously doesn't work if I need to test refinery changes in which case I pass custom refinery jars [13:55:28] sorry, *test refinery-source changes [13:55:46] I think everyone does this their own way though [13:55:58] got it makes sense [13:56:02] will take notes for the next time [14:02:35] 10Analytics, 10Reading Depth, 10Readers-Web-Backlog (Tracking): [Bug] Many ReadingDepth validation errors logged - https://phabricator.wikimedia.org/T216063 (10Milimetric) These failures are expected, though, the spec does not account for such prefixes, it just expects an encoded JSON blob. Is this somethin... [14:02:45] (03CR) 10Elukey: [C: 04-1] "Luca, how dare you not using the closing tag! This will not work, please study oozie before doing silly things! :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:06:55] ok after fixing two errors it seems running [14:08:09] (03PS2) 10Elukey: projectview: move to oozie workflow to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) [14:08:21] arg, sorry I missed the tag in my review [14:10:23] also s/hive_jdbc_url/hive2_jdbc_url/g [14:10:29] my bad :( [14:13:57] (03CR) 10Elukey: "https://hue.wikimedia.org/oozie/list_oozie_workflow/0017560-190626064919032-oozie-oozi-W completed!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:14:25] milimetric: https://hue.wikimedia.org/oozie/list_oozie_workflow/0017560-190626064919032-oozie-oozi-W completed, all green [14:15:14] elukey: sweet. if you wanna be super paranoid like me query both that and the prod version of it and do a count(*) or something like that [14:15:50] milimetric: if you want go ahead, it shouldn't be necessary but better safe :) [14:16:03] year=2019/month=7/day=12/hour=0 [14:16:21] I can count* now [14:18:28] 56024 in both :) [14:18:30] huh... there is one slightly weird thing... [14:18:49] nvm, it was just my terminal formatting things poorly [14:18:51] ok, all good [14:19:40] (03CR) 10Elukey: [V: 03+2] "select count(*) for the same hour in the elukey and wmf databases is the same number!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:19:52] milimetric: do you know how el per page and per session sampling works? [14:20:08] what is the algorithm? [14:20:12] or...ok looking for code... [14:22:57] thanks milimetric! [14:23:37] ottomata: yeah, there are util functions in the client [14:24:13] ottomata: sampling functions: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging/core.js#L211-L236 [14:24:26] ya found, it confused me because of the hex string stuff... [14:24:28] but i think i get it [14:24:34] the slice 8 is beacuse we only accept 32 bit hex strings [14:24:38] which can be max 8 chars long [14:24:41] but the trickiest part are the nuances of mw.user.sessionId() and mw.user.getPageviewToken() [14:24:48] that's fine [14:24:54] i just wanted to see the sample alg [14:25:18] we were thinking of sampling rates as 0.0 - 1.0 [14:25:22] not 'population size' [14:32:58] (03PS1) 10Milimetric: Improve examples in docs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522480 [14:33:08] (03CR) 10Milimetric: [V: 03+2] Improve example run command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520661 (owner: 10Milimetric) [14:33:36] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Improve examples in docs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522480 (owner: 10Milimetric) [14:33:51] milimetric: i had a stream config idea, want to brain bounce with you when you have a min [14:34:02] to the cave, ottomata [14:35:04] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) Recap of what it has been done so fare in various (sub) tasks:... [14:53:17] (03PS6) 10Fdans: Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) [15:03:47] ping ottomata : swift meeting [15:06:21] (03PS2) 10Fdans: [wip] Add file extension and media classification to mediacounts job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522390 (https://phabricator.wikimedia.org/T225911) [15:18:09] what/! [15:18:09] AGHWHAT [15:18:20] ottomata: ahem ya [15:18:29] i did not see that mEETING YARGH [15:18:36] NOR did I get a reminder [15:18:37] GRRRR [15:18:42] (03CR) 10Nuria: [C: 03+2] Most special pages should not be pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730) (owner: 10Nuria) [15:20:08] yargh [15:20:17] sorry nuria [15:20:56] was here, but my headphones are plugged in, but i'm not wearing them [15:21:02] so i didn't get any sound pings [15:21:03] GROWL [15:22:53] ayayay [15:23:12] let's talk about sampling if you want before standup [15:24:16] nuria milimetric anyone against classifying stl (3d imagery) as image in mediacounts? [15:24:32] fdans: stl? [15:24:37] I'm adding a couple missing file types I found testing [15:24:48] nuria: https://en.wikipedia.org/wiki/STL_(file_format) [15:25:26] I say either image or data [15:26:05] nuria: my sample of one hour showed 178 hits on stl files so... might as well add it to the classifier [15:27:12] (03PS5) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [15:27:20] (03CR) 10jerkins-bot: [V: 04-1] Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [15:27:52] fdans: i see, what would be the process of adding a new filetype? [15:28:53] nuria: adding it to the KnownTypes enum [15:29:59] the classification wouldn't have a retroactive effect though [15:30:49] fdans: right, taht makes sense [15:31:06] nuria: although I guess if there was a big file type that we have missed we could backfill only for files with that extension [15:31:26] fdans: and when we load to cassandra are we going to explode by file type counts? [15:31:40] I don't think so [15:32:30] I think media classification for sure, but file type is a bit too much [15:33:53] nuria: here's a question: should we save extension pairs (MID/MIDI, JPG/JPEG, TIF/TIFF) as the same extension? [15:34:15] this is how the file type distribution looks like for one hour: [15:34:23] https://www.irccloud.com/pastebin/7gnIFyqC/ [15:34:50] (03CR) 10Nuria: "Let's please also verify that data (not only counts) looks the same doing some spot checking for values that come from refine like geo/pro" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [15:35:46] fdans: ok, media classificantion seems fine to explode by but I agree file type is too much [15:36:40] (03PS1) 10Ottomata: oozie swift/upload - add support for X-Delete-After header [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522503 (https://phabricator.wikimedia.org/T213976) [15:37:31] nuria: what about extension pairs? [15:38:02] fdans: I think media type is sufficient (we should document this on wikitech if we agree) [15:38:25] fdans: for exploding /storing in cassandra that is [15:39:16] nuria: yeayea but what I mean is right now we're counting jpg and jpeg separately, I'm wondering if they should be counted as one file type [15:39:32] 10Analytics, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Ottomata) [15:40:59] fdans: ah, sorry, i missunderstood [15:41:12] (03CR) 10Nuria: [C: 03+1] "Looks good, adding erik in case he has any suggestions" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522503 (https://phabricator.wikimedia.org/T213976) (owner: 10Ottomata) [15:41:52] fdans: it seems that those should be consolidated but that will change current file generation right? [15:42:46] fdans maybe let's talk about it at standup? [15:42:48] nuria: the file name wouldn't change, but we would consolidate the file_type column to only one type [15:47:00] fdans: seems that that makes total sense, yeah [15:47:44] (03CR) 10Nuria: "Thanks for doing this" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522480 (owner: 10Milimetric) [15:52:28] (03CR) 10Fdans: [V: 03+1] "This change has been tested on the cluster successfully" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [15:53:12] fdans: to your original question i think adding stl seems a bit much [15:53:33] fdans: maybe we need an other category that matches several file types this being one of them? [15:53:42] (03CR) 10Elukey: [V: 03+2] "> Let's please also verify that data (not only counts) looks the same" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [15:53:53] nuria: yes, there's an "other" file cat [15:53:58] that's the default classification [15:54:10] fdans: then i would keep this file there [15:54:11] for one hour: [15:54:13] https://www.irccloud.com/pastebin/ThGvMrp1/ [15:56:11] (03PS7) 10Fdans: Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) [16:00:18] ping milimetric standduppp [16:20:21] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Nuria) Let's figure out how to deploy only what we need to notebook hosts [16:29:10] (03CR) 10Elukey: [V: 03+2] "Dumped the hour from the elukey db and the wmf db in two separate files, compared with diff, all good." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [16:49:49] * elukey off! [16:50:15] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10Krinkle) Yes, I mostly changed our practice of deleting mirrors to archiving them (which means it's fully ready-only, including no pull requests or other sources of notifica... [17:33:19] (03PS6) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [17:33:30] (03CR) 10jerkins-bot: [V: 04-1] Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [17:34:56] (03PS7) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [17:35:21] (03CR) 10jerkins-bot: [V: 04-1] Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [17:38:58] hey a-team, qq: there is folder on the stat machines that allows to share files via a public URL, which was that? [17:42:29] dsaez: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Dashiki#How_to_get_your_data_available_via_http [17:44:39] thanks nuria! [17:57:26] (03PS8) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [17:59:36] (03PS9) 10Nuria: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) [18:06:50] milimetric [18:06:51] oh [18:06:52] https://wikitech.wikimedia.org/wiki/MediaWiki_and_EtcdConfig [18:06:52] hm [18:08:40] 10Analytics, 10WMDE-Analytics-Engineering: Public Data Review Needed - https://phabricator.wikimedia.org/T227905 (10GoranSMilovanovic) [18:09:30] 10Analytics, 10WMDE-Analytics-Engineering: Public Data Review Needed - https://phabricator.wikimedia.org/T227905 (10GoranSMilovanovic) [18:11:14] 10Analytics: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [18:12:28] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [18:13:11] milimetric: would appreciate a look over of ^ before I add the tech com rfc tag [18:13:18] also hip ^ [18:16:39] ottomata: we can merge this one right? (sorry about the multiple late patches i totally had forgotten about how to correct history in gerrit patches that are on top of other already closed patches) [18:17:20] yes we can merge! [18:20:19] (03CR) 10Nuria: [C: 03+2] Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [18:25:08] (03Merged) 10jenkins-bot: Refactoring eventlogging-specific hostname check [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/521577 (https://phabricator.wikimedia.org/T227150) (owner: 10Nuria) [18:25:36] (03CR) 10Nuria: "Let's merge this job and monitor it a bit after merge before changing others, sounds good?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [18:25:38] (03CR) 10Nuria: [C: 03+2] projectview: move to oozie workflow to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522395 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [18:26:40] ottomata: it makes me think about the recent move away from downloading schema modules from the EL client. We did that for performance reasons and this feels like it would be a half step back [18:27:02] milimetric: ? [18:27:07] since it would trigger an extra load on maybe the majority of page loads [18:27:13] yeah it would [18:27:23] don't know how else to implement this tho? [18:27:53] so I was trying to brainstorm ways to maybe sync the config periodically somewhere that php has access to [18:28:11] ? [18:28:49] the rest looks good, I would file it this way and I can add a note to that effect, that we’re aware of the potential perf problem and are looking for some solutions that maybe others could benefit from [18:28:57] ok [18:29:42] like maybe you edit the meta page and every 5 minutes if there are changes it updates a php file that’s read by mediawiki when rendering the page with instrumentation [18:29:59] milimetric: but that doesn't help the extra round trip, right? [18:30:14] ottomata, milimetric : the perf hit can be avoided if the info from that page is passed on the 1st response to mw as part of the page [18:30:32] yeah, ottomata what nuria said [18:30:38] hm, nuria that is very possible, maybe even more so with https://wikitech.wikimedia.org/wiki/MediaWiki_and_EtcdConfig [18:30:40] ? [18:30:43] that’s the idea, to somehow do that [18:30:54] i see [18:31:42] if you render the config to a php file that’s deployed on the app servers, they would render the page. Of course, caching would slow the changes [18:31:51] ottomata, milimetric: I do not think the idea of having the client fetching a "meta" page before executing an event will go through [18:32:03] ottomata, milimetric : perf wise, as we just moved away from that [18:32:04] yeah, it probably won’t [18:32:18] 'Whenever a MediaWiki process starts, MediaWiki tries to fetch the config data from the local cache (APC for fastcgi/web requests, a local hash on cli); if it's not there, or it's stale, it will try to fetch fresh data from the Etcd cluster. ' [18:32:19] ottomata, milimetric : so we should probably not even consider it [18:32:45] nuria: so that sounds like a mediawiki solution is the only way to go [18:32:45] ottomata: that is for "state" not "config" right? (major disclaimer on page) [18:32:54] ottomata: i mean /etcd [18:32:57] I think it’s decoupled, is what I’m saying. Where you edit the config and how it gets to the client [18:33:24] milimetric: ya +1, we can talk in batcave for 10 mins if you want [18:33:28] nuria: hm, it seems the line is blury. [18:33:33] i'd bc [18:33:34] now? [18:33:36] config and how it ends up on the client [18:33:39] bc sure [18:33:53] k [18:56:06] 10Analytics, 10Operations, 10ops-eqiad: Broken disk on analytics1072 - https://phabricator.wikimedia.org/T226467 (10Cmjohnson) You have successfully submitted request SR994463766 is the Dell ticket created. I did see the disk in megacli so I am not sure the TSR report I sent them will include the disk. I d... [18:57:32] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [19:12:45] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [19:13:09] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [19:17:44] (03PS1) 10Ladsgroup: Track number of external identifiers on wikidata alongside statements [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/522537 (https://phabricator.wikimedia.org/T224129) [19:30:24] (03PS1) 10Ottomata: Deploy to notebook hosts as a separate environment to avoid deploying artifacts [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/522542 (https://phabricator.wikimedia.org/T227132) [19:35:18] (03PS2) 10Ladsgroup: Track number of external identifiers on wikidata alongside statements [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/522537 (https://phabricator.wikimedia.org/T224129) [19:35:39] (03CR) 10jerkins-bot: [V: 04-1] Track number of external identifiers on wikidata alongside statements [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/522537 (https://phabricator.wikimedia.org/T224129) (owner: 10Ladsgroup) [19:36:42] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Krinkle) [19:37:16] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Krinkle) I've changed `mw.config` to "a ResourceLoader module". Principally the same from JS perspective, but reflecting current best practices. [19:51:23] milimetric: timo gave me https://www.mediawiki.org/wiki/ResourceLoader/Package_modules#Data_and_config_bundling [20:07:00] (03PS3) 10Ladsgroup: Track number of external identifiers on wikidata alongside statements [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/522537 (https://phabricator.wikimedia.org/T224129) [20:07:03] (03PS1) 10Ladsgroup: Fix queries to follow the new actor schema [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/522549 (https://phabricator.wikimedia.org/T227788) [20:13:24] ottomata: that’s pretty cool, did you look into the caching/deploying questions? I didn’t see details on that page but I skimmed [20:20:02] the deployment stil lneeds worked out [20:20:11] so all the things we talked about for config storage are relevant [20:20:23] ResoureLoader is just what we'd use to get the stuff into mw.config [20:20:35] likley via a custom callback that fetches all the config from somewhere.. [20:20:40] git repo http server, JsonConfig, something [20:39:02] (03CR) 10Nuria: [C: 03+2] "Nice! Let's update docs about refinery deployment" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/522542 (https://phabricator.wikimedia.org/T227132) (owner: 10Ottomata) [20:50:12] 10Analytics, 10EventBus: [WIP] RFC: Stream Configuration Service - https://phabricator.wikimedia.org/T227906 (10Ottomata) [20:58:34] (03CR) 10Nuria: [C: 04-1] "I realize that these are quite a few comments, I can also push a patch to perhaps better explain what I mean with code." (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans)