[01:30:01] 10Analytics, 10Performance-Team, 10Vue.js: Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Niedzielski) [01:36:26] PROBLEM - Hadoop NodeManager on an-worker1084 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [01:48:16] PROBLEM - Hadoop NodeManager on an-worker1085 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [05:34:05] !log restart hadoop-yarn-nodemanager on an-worker108[4,5] - failed after GC OOM events (heavy spark jobs) [05:34:12] !log restart hadoop-yarn-nodemanager on an-worker108[4,5] - failed after GC OOM events (heavy spark jobs) [05:34:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [05:34:20] RECOVERY - Hadoop NodeManager on an-worker1084 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [05:34:56] RECOVERY - Hadoop NodeManager on an-worker1085 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [06:34:50] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10elukey) p:05Triage→03High [06:36:44] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10elukey) One of the issues seems to be: ` 1.1T atskafka_test_webrequest_text-0 1.1T atskafka_test_webrequest_text-1 ` We currently only send traffic to that topic from a single cp3xxx host,... [06:38:57] * elukey errand for a bit [10:21:38] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10elukey) Jumbo1002: ` 12G eventlogging_PaintTiming-0 19G eventlogging-valid-mixed-0 19G eventlogging-valid-mixed-1 19G eventlogging-valid-mixed-6 19G eventlogging-valid-mixed-7 19G eventlogg... [10:41:40] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10elukey) Overall I have the following ideas: * short term, we could drop the atskafka test topic to free space, and re-create it with more partitions to spread the load among multiple broker... [11:30:58] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10elukey) p:05High→03Medium /srv usage is now around 83% after dropping the ats test topic! [11:34:06] heya elukey :] can I create a keytab for my user in stat1007? Is there any conventions or security measures I should take? It's for the airflow instance. [11:35:02] mforns: hey! So we do not create keytabs for users, in theory if you need one you can use the analytics-privatedata user [11:35:06] for testing I mean [11:37:56] or even better, you could move your testing to an-launcher1001 [11:38:00] and use the analytics user keytab [11:38:03] mforns: --^ [11:39:09] elukey: yes, but I need to pip install stuff with the same user than I execute airflow... [11:39:23] no? [11:39:31] and analytics user does not have pip [11:40:17] mforns: in theory you can install everything as your user, then set up airflow to run with the user that you want [11:40:24] if you want we can batcave, and I show what I'm doing to run airflow [11:40:32] sire [11:40:40] ok, omw [11:45:37] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10nshahquinn-wmf) It looks like #product-analytics is tagged just for information. If this is a request for us to do something, please m... [12:13:23] (03CR) 10Joal: "Comments inline and:" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [12:37:17] (03PS6) 10Joal: Update hive geocoded-data udf [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 [12:41:15] Hi elukey - let's sync when you have time on the hadoop worker issue [12:43:37] joal: sure, I am with Marcel now, and after I'd need to go buy some groceries, ok if later on? [12:43:54] no prob elukey :) ping me when you want :) [12:49:42] ack thanks :) [12:49:47] * elukey errand! groceries [13:38:47] 10Analytics, 10Operations: kafka-jumbo1003 /srv disk space usage over 90% - https://phabricator.wikimedia.org/T250347 (10Ottomata) Hm. Moving partitions is a bit annoying but is possible. https://docs.cloudera.com/runtime/7.1.0/kafka-managing/topics/kafka-manage-cli-reassign-overview.html I'd also be ok wi... [13:45:30] (03CR) 10Ottomata: "> > Could we make each of these a impls a singleton?" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [13:49:00] (03PS1) 10Bearloga: Add README [analytics/wmf-product] - 10https://gerrit.wikimedia.org/r/589307 (https://phabricator.wikimedia.org/T230743) [13:49:38] (03CR) 10Bearloga: [V: 03+2 C: 03+2] Add README [analytics/wmf-product] - 10https://gerrit.wikimedia.org/r/589307 (https://phabricator.wikimedia.org/T230743) (owner: 10Bearloga) [13:58:36] (03CR) 10Ottomata: Update hive geocoded-data udf (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [14:01:08] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) FYI plans changing slightly in https://phabricator.wikimedia.org/T238230#6059974. I'll keep... [14:17:30] 10Analytics, 10Analytics-Kanban: Delete raw events for mediawiki_ , the refined data is kept indefinately - https://phabricator.wikimedia.org/T245126 (10Ottomata) Ah, we did this already as part of {T245124}. This is done. [14:17:34] 10Analytics, 10Analytics-Kanban: Delete raw events for mediawiki_ , the refined data is kept indefinately - https://phabricator.wikimedia.org/T245126 (10Ottomata) 05Open→03Resolved [14:18:25] 10Analytics, 10Analytics-Kanban, 10EventStreams: KafkaSSE issue tracker and continuous integration - https://phabricator.wikimedia.org/T248044 (10Ottomata) > However, we do generally keep at least a mirror of these in Gerrit I'm not sure this is true. E.g I'm not finding https://github.com/wikimedia/service... [14:24:04] joal: i am back if you have time [14:24:22] Hi elukey :) [14:24:25] batcave? [14:24:59] sure [14:40:05] (03CR) 10Nuria: [C: 04-1] Update hive geocoded-data udf (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [14:44:38] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Nuria) >Once the schema, refinery, and eventgate-wikimedia changes are in place, all we have to do for SearchSatisfaca... [14:46:32] (03CR) 10Ottomata: Update hive geocoded-data udf (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [15:20:07] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Ottomata) Ping @Isaac how goes, let us know when we can re-enable deletion. [15:24:55] joal: from https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&from=now-24h&to=now&var-datasource=eqiad%20prometheus%2Fanalytics&var-hadoop_cluster=analytics-hadoop&var-worker=an-worker1085 it is more clear to see what happened [15:25:04] an-worker1085 was one of the nm that failed [15:25:05] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Isaac) @Ottomata thanks for the ping -- I have one final adjustment to make to the query but then we will start loading in data from January 1st onwards. Each day takes a few hours to run the... [15:25:15] see old gen? spiking up to 3.6G over 4.0 [15:25:27] and I bet the precision of those metrics is not great [15:25:28] yup [15:25:46] also, very interesting, gc time [15:25:49] you obviously were right elukey about old-gen :) [15:26:08] see that young gen also bumps up? [15:26:27] It may be too much pressure on eden [15:26:42] that would make sense, a lot of new things (temporary) to allocate to support the shuffling [15:35:39] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Nuria) @Isaac Also please be so kind to document where results of this query are stored (should be a location on hdfs) and it will be wise to put query in source control. [15:50:07] 10Analytics, 10ContentTranslation, 10Language-Team (Language-2020-Focus-Sprint): Test Performance of Marian NMT translation in stat cluster - https://phabricator.wikimedia.org/T247245 (10JAllemandou) Hi @santhosh, I have thought of the project and I have a question for you: Have you tested that multiple mode... [15:50:12] elukey: --^ [15:52:04] yo mforns joal 3 of the 4 changes in the deployment train etherpad aren't merged [16:04:56] 10Analytics, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Milimetric) p:05Medium→03High marking high so we get to it before everyone forgets the context (still might be a while) [16:15:41] Joal: this looks good https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/BotDetection, will work more on it today [16:21:54] so no grooming today or are we doing it? [16:22:00] (sorry didn't get it) [16:22:17] I know that there is the tuning session but it seems that my browser crashes badly with zoom :D [16:25:41] 10Analytics, 10Product-Analytics: Can't publish my draft dashboard on superset - https://phabricator.wikimedia.org/T248904 (10Milimetric) Seems related to a recently merged pull request (https://github.com/apache/incubator-superset/pull/9279) but I asked to verify. It seems very strange that this button is br... [16:28:09] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Isaac) > Also please be so kind to document where results of this query are stored (should be a location on hdfs) and it will be wise to put query in source control. @Nuria sounds good -- be... [16:49:29] 10Analytics, 10Research: Check home/HDFS leftovers of flemmerich - https://phabricator.wikimedia.org/T246070 (10Isaac) @elukey -- thanks for the ping. I'll aim for completing this before the holidays next week. Also, copying over from an email conversation for completeness and so I don't lose this information... [16:54:48] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Nuria) >I'm happy to work on a better solution if you'd like, but for now I'll just keep that consistent with what's on SWAP. swap + github sounds good, nice [17:05:28] elukey: no grooming [17:05:38] ack! [17:06:20] joal: A question for when you're around, how does mediawiki_history revert detection work for multi-content revisions? Does it use checksums of the wikitext only? I've been talking with the Structured Data team, and we're wondering if we can detect reverts of SD edits. [17:08:39] Nettrom: we cannot [17:08:56] Nettrom: as reverts are only on checksums [17:09:03] Nettrom: that is a perfect case for event data [17:09:41] ottomata: A question for when you're around: last week we found events missing in Sept 2019 from several EventBus streams, and it seems related to https://phabricator.wikimedia.org/T233718 The phab task mentions backfilling data, but I'm not sure that was done for all streams. I'm wondering if there's some documentation of this on Wikitech, would be good to have to reference in the Product Analytics team. [17:10:38] Nettrom: more so cause the MCR work might change significantly at the db level as we know there are many scaling issues [17:10:47] nuria: ah, so the SHA1 checksum is computed on the wikitext and not anything else. Agree that having event data around this would be useful, I think it would help us understand property usage as well [17:11:59] Nettrom: more so cause i think the db as is - if structure data grows significantly - will change very significantly (the estimate is that it can only be as is for ~1 more year due to scaling concerns, 1 year might be too little) [17:12:36] 10Analytics: Superset: "Error while fetching database list" - https://phabricator.wikimedia.org/T249825 (10elukey) Ok so in https://phabricator.wikimedia.org/T249923 I should have solved the problem that Andre was facing. I also see the following: ` AH01617: user srishakatux: authentication failure for "/": P... [17:15:20] nuria: Ah, I see, makes sense that there'll be changes to reduce space usage then. Sounds like this is something we in the PA team should think about too, because if reverts don't make sense on Commons it renders mediawiki_history less useful [17:15:32] it's not trivial how to make them work, though [17:15:42] Nettrom: ya, this must be event data [17:16:00] Nettrom: SD has no data flows of any sort and it shows in issues such as these [17:16:24] * elukey off! [17:18:57] Nettrom: i think build the data flows in this case would not be hard but it needs to be done by the platform teams, i think filing a ticket is a good place to start [17:25:47] ottomata: Don't wait for me with core-platform, I can't make it this evening [17:27:00] joal: can talk about geo udf now? [17:35:27] k no one showed up anyway :) [17:35:36] nuria: i can talk if joal can but i think we need joal [17:36:27] nuria: actually just discovered that there's a dist dir for languages in numbro with minified locales, so we can split with that [17:48:14] nuria, ottomata: ready! [17:50:04] arf - too late it seems :( [17:52:35] nuria, Nettrom - Some precision on mediawiki-history: we use the revision-sha1, which is IIRC the checksum of checksums - Therefore a revert in structured-data is seen as a revert, but it also means that reverts are identified that are not text-only reverts [17:53:29] nuria, Nettrom: We will in any case need to be able to update mediawiki-history so that the semantics of a revert stays at pure text-level [17:58:53] Nettrom: Please let me know if what I said above makes sense to you :) --^ [18:08:40] (03PS1) 10Lex Nasser: Allow pageview titles that include Unicode character values above 0xFFFF like emoji [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) [18:10:53] ottomata: just curious, can I query the superset metadata? [18:11:11] like, the mysql/whatever db it's in? [18:12:55] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10lexnasser) Just submitted a patch with the fix and some new tests: https://gerrit.wikimedia.org/r/589383 --- @Nuria Confirmed... [18:13:14] mysql://superset_staging@an-coord1001.eqiad.wmnet/superset_staging [18:15:11] hm, I'm just getting access denied, good to know [18:18:55] Nettrom: see joal [18:19:25] comment, i was mistaken as i though the problem was reverts were only text reverts, ya [18:24:39] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) some malicious pageview titles here: {T144100} and here: {T152628} Some of these can probably be added to unit tests [18:25:07] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10hueitan) a:03hueitan [18:25:10] joal: I will be free in 1 hr again [18:25:19] nuria: ack - I'll be there [18:25:37] nuria: adding graphs to the bots-doc in the meantime [18:26:13] joal: k, i will do some edits there today too [18:30:24] joal: thanks for the insight, that makes sense to me, both that we're currently identifying non-text reverts, and that'd be interested in updating mediawiki_history so the semantics stay [18:31:19] ack Nettrom :) [18:46:46] 10Analytics, 10Product-Analytics, 10wmfdata-python: wmfdata cannot recover from a crashed Spark session - https://phabricator.wikimedia.org/T245713 (10kzimmerman) Per conversation with Neil, Moving to the backlog since this hasn't come up as a major issue [18:49:42] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 2 others: EventLogging MEP Upgrade - https://phabricator.wikimedia.org/T238544 (10jlinehan) [18:51:17] 10Analytics, 10Analytics-Kanban: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Milimetric) [18:52:10] (03CR) 10Mforns: Configure geoeditors monthly public Oozie job to work with geoeditors public monthly table (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser) [18:53:42] 10Analytics, 10Analytics-Kanban: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Milimetric) Just being cautious, I saved the databases I deleted as a yaml export {F31759952}. FYI: I had to delete some untitled queries and things from Joseph that were attached to two of the dbs.... [18:57:13] anyone wanna brainbounce with me on https://phabricator.wikimedia.org/T248560 ? [18:58:50] milimetric: looking [18:58:54] milimetric: I can help if you wish [18:59:48] I dropped it a while back and it's fairly important, without re-consuming from Kafka we're losing data, and even so we're losing stuff past the buffer [19:00:01] I don't think both of you need to look though, I just need someone to generate ideas with [19:00:02] hm [19:00:18] i'll help milimetric my head has been in both of those Ua parser places recently too [19:00:23] k [19:00:29] ack [19:00:41] in bc? [19:01:29] omw [19:08:39] (03CR) 10Awight: "Great idea! The patch looks good, just needs a bit more minimization." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) (owner: 10Lex Nasser) [19:27:29] actually milimetric instead of doing all that i just said, let's just migrate InukaPageView to the new system asap [19:27:42] that way we don't have to make any eventlogging-processor changes [19:27:50] up to you [19:27:53] and neil [19:29:11] joal: free if you guys want to talk [19:29:20] ok! [19:29:21] ready :) [19:29:21] joal: ? [19:29:40] bc [19:30:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Product-Analytics: EventLogging does not properly classify KaiOS user agents - https://phabricator.wikimedia.org/T248560 (10Ottomata) It looks like our python-ua-parser debian package deployed on eventlog1002 is old. I just had a quick go at rep... [19:34:33] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Product-Analytics: EventLogging does not properly classify KaiOS user agents - https://phabricator.wikimedia.org/T248560 (10Milimetric) Andrew figured out the problem, he'll explain separately along with options to fix (basically we forgot to upd... [19:55:35] 10Analytics, 10VPS-project-codesearch: Add analytics/* gerrit repos to code search - https://phabricator.wikimedia.org/T249318 (10Milimetric) Ok, easy enough, putting the list in the description so others can annotate it. I'm putting (useless) next to the repos I don't think we should include. Some of the ar... [19:55:44] (03PS7) 10Joal: Update hive geocoded-data udf [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 [19:56:04] nuria, ottomata: I made the methods public but also kept the static variables public as they are used statically in the UDF code [19:56:25] (and in java, no abstract can be static ...) [20:04:42] 10Analytics, 10VPS-project-codesearch: Add analytics/* gerrit repos to code search - https://phabricator.wikimedia.org/T249318 (10Milimetric) [20:27:15] 10Analytics, 10Analytics-Kanban: Test aqs_hourly job from (Search team's) Airflow - https://phabricator.wikimedia.org/T248328 (10mforns) @EBernhardson In the end, I preferred to create an Airflow instance under my user in stat1007, because of the multi-tenant issue you mentioned. Nevertheless, your tips, code... [20:29:44] 10Analytics, 10Analytics-Kanban: Test aqs_hourly job from Airflow testing instance - https://phabricator.wikimedia.org/T248328 (10mforns) [20:44:33] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10observability: Validation error for invalid value type should include property name - https://phabricator.wikimedia.org/T116719 (10Ottomata) I won't be fixing eventlogging-processor for this, but EventGate should do better. I'm repurposing this... [20:45:35] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10observability: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Ottomata) [20:46:40] 10Analytics, 10Analytics-Kanban, 10EventStreams: KafkaSSE issue tracker and continuous integration - https://phabricator.wikimedia.org/T248044 (10Ottomata) 05Open→03Resolved Resolving for now as a few of the items have been resolved, and I'm not sure we mirror from github to gerrit. [20:47:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) [20:47:49] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) [20:49:55] 10Analytics, 10Analytics-Kanban: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Ottomata) We should also delete the EventLogging MySQL database, as it no longer exists. [20:51:25] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10observability: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Ottomata) [21:02:22] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10lexnasser) @Nuria Thanks for the suggestions! T144100 is an issue later in the Analytics pipeline, so I don't think a unit te... [21:19:33] (03PS2) 10Lex Nasser: Allow pageview titles that include Unicode character values above 0xFFFF like emoji [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) [21:25:19] (03PS3) 10Lex Nasser: Allow pageview titles that include Unicode character values above 0xFFFF like emoji [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) [21:27:38] (03PS4) 10Lex Nasser: Allow pageview titles that include Unicode character values above 0xFFFF like emoji [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) [21:30:01] (03PS5) 10Lex Nasser: Allow pageview titles that include Unicode character values above 0xFFFF like emoji [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) [21:32:11] (03CR) 10Lex Nasser: "Sorry for all the patch sets, test cases not relevant to this task kept being changed, making the diff a little too extensive. Should be a" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) (owner: 10Lex Nasser) [21:33:47] (03CR) 10Awight: [C: 03+1] "Very nice!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/589383 (https://phabricator.wikimedia.org/T245468) (owner: 10Lex Nasser) [23:22:40] (03PS6) 10Lex Nasser: Configure geoeditors monthly public Oozie job to work with geoeditors public monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) [23:28:43] (03CR) 10Lex Nasser: "Mforns, thanks for the CR!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser)