[00:42:54] 10Analytics, 10MediaWiki-extensions-ORES, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Later), and 4 others: ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10mobrovac) [00:55:22] 10Analytics, 10EventBus, 10Product-Analytics, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10mobrovac) [06:20:45] morning! [06:20:55] chelsyx: o/ are you around by any chance? [06:21:12] I can see a huge job in hadoop that is consuming a ton of resources [06:22:29] that doesn't seem to be running from what I am seeing [06:22:34] but the resources are allocated [06:24:08] !log kill of application_1555511316215_18282 on Hadoop due to excessive resource usage [06:24:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:24:24] Going to send an email to chelsyx, really sorry but the vcores were exhausted :( [06:31:40] elukey: o/ [06:37:33] fdans: morning! [07:05:26] ah lovely [07:05:30] jobs killed sigh [07:05:32] elukey: nvm the recent alerts [07:05:37] that's me [07:06:06] !log restarted webrequest bundle [07:06:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:06:27] fdans: ah ok that was weird :D [07:07:04] there are two webrequest load bundles running though [07:07:06] is it expected? [07:07:08] elukey: if you needed a bit of a jump to start the day with energy [07:08:23] sorry elukey, I thought I killed the bundle but it was the coord, now the prior bundle is killed [07:08:58] fdans: super [07:24:11] I'm looking at the failed jobs now [07:26:10] they seems failing in refine [07:26:43] converting to local hdfs://analytics-hadoop/wmf/refinery/2019-03-27T21.09.13+00.00--scap_sync_2019-03-27_0001-dirty/artifacts/org/wikimedia/analytics/refinery/refinery-hive-0.0.87.jar [07:26:47] Failed to read external resource hdfs://analytics-hadoop/wmf/refinery/2019-03-27T21.09.13+00.00--scap_sync_2019-03-27_0001-dirty/artifacts/org/wikimedia/analytics/refinery/refinery-hive-0.0.87.jar [07:26:51] Intercepting System.exit(1) [07:26:54] Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [1] [07:26:57] fdans: --^ [07:27:22] did you upload refinery to hdfs? [07:27:47] I mean after the last deployment attempt [07:28:13] no, killing, doing that, restarting [07:28:14] >.< [07:28:16] sorry [07:30:51] that is super fine, it is only some spam in the emails, I am a champion of triggering alerts :D [07:31:32] you-re too nice elukey [07:43:04] !log refinery uploaded to hdfs and webrequest bundle restarted [07:43:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:02:26] joal: when you are online and have time, I am wondering why we use the fair scheduler in yarn as opposed to the capacity scheduler.. I am looking for a way to limit users (max vcores/memory etc...) and the capacity scheduler seem really nice [08:02:34] but I am really ignorant about the subject [08:26:33] elukey: there's some data loss warnings, I'm looking into them [08:38:11] fdans: ack thanks! [09:37:07] (03PS18) 10Elukey: Add artifacts for Debian Buster and upgrade to 0.32rc2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) [10:01:54] (03PS19) 10Elukey: Add artifacts for Debian Buster and upgrade to 0.32rc2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) [10:40:35] * elukey lunch + errand! [11:16:59] 10Analytics, 10Research-management: Test GPUs with an end-to-end training task (Photo vs Graphics image classifier) - https://phabricator.wikimedia.org/T221761 (10Miriam) [12:24:57] (03PS10) 10Mforns: Add edit_hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501197 (https://phabricator.wikimedia.org/T220092) [12:27:25] (03CR) 10Mforns: [C: 04-2] "Thanks for the +2s." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501607 (https://phabricator.wikimedia.org/T211173) (owner: 10Mforns) [12:29:30] (03PS5) 10Mforns: Add edit_hourly to list of tables to be purged of old snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501328 (https://phabricator.wikimedia.org/T220092) [12:29:55] (03PS6) 10Mforns: Add oozie job to load edit_hourly to druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501607 (https://phabricator.wikimedia.org/T211173) [12:30:03] (03CR) 10Mforns: [C: 04-2] Add oozie job to load edit_hourly to druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501607 (https://phabricator.wikimedia.org/T211173) (owner: 10Mforns) [13:45:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10elukey) Very good news, it seems that the Superset project has finally found a way to release under Apache license. I upgraded today to 0.32rc2 (th... [13:51:45] (03PS11) 10Mforns: Add edit_hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501197 (https://phabricator.wikimedia.org/T220092) [13:59:00] nuria: you will be delighted to know that your webrequest histogram is broken on 0.32 [13:59:03] lol [14:01:06] 10Analytics, 10Analytics-Kanban: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608 (10Halfak) Hey folks. I've been following this task, but I might not have the full context, so take what I say with a grain of salt that is appropriately sized. "user_text" is a co... [14:43:35] 10Analytics, 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Core Platform Team (Modern Event Platform (TEC2)), and 4 others: ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10Ladsgroup) [14:44:11] 10Analytics, 10Wikimedia-Stream: Update EventBus RCFeed config to use newly refactored settings - https://phabricator.wikimedia.org/T158106 (10Ottomata) 05Open→03Resolved Done in another task [14:44:14] 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651 (10Ottomata) [14:44:50] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) [14:44:52] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventGate should be able to configure hasty and guaranteed kafka producers individually - https://phabricator.wikimedia.org/T219032 (10Ottomata) 05Open→03Resolved [14:55:24] 10Analytics, 10Analytics-Cluster, 10Operations: furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error - https://phabricator.wikimedia.org/T221483 (10Ottomata) [14:55:27] 10Analytics, 10Analytics-Cluster, 10Operations: Remove Hadoop configs and unmount /mnt/hdfs from unused backup hosts (furud, +) - https://phabricator.wikimedia.org/T221629 (10Ottomata) 05Open→03Declined Oh, actually, /mnt/hdfs is not puppetized. It was leftover from when it was. I just removed it from f... [14:56:11] 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731 (10Ottomata) Can we close this task? [14:56:37] 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651 (10Ottomata) [14:56:44] 10Analytics, 10Wikimedia-Stream, 10Patch-For-Review: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748 (10Ottomata) 05Open→03Declined {T219552} is better [14:57:51] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088 (10Ottomata) [14:57:54] 10Analytics, 10ChangeProp, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Support per-topic configuration in EventBus service - https://phabricator.wikimedia.org/T157092 (10Ottomata) 05Open→03Declined I don't think this is really needed. If it is, it will be part of {T205319} [14:58:36] 10Analytics-Kanban, 10EventBus, 10Operations, 10netops: Allow analytics VLAN to reach schema.svc.$site.wmnet - https://phabricator.wikimedia.org/T221690 (10Ottomata) a:05Ottomata→03None [14:59:44] 10Analytics, 10Analytics-EventLogging: Increase number of partitions of eventlogging-client-side topic in Kafka jumbo-eqiad - https://phabricator.wikimedia.org/T205436 (10Ottomata) 05Open→03Declined [15:00:48] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Make schemas use required $schema property with absolute path (not absolute URL) to the schema - https://phabricator.wikimedia.org/T208361 (10Ottomata) [15:00:51] !log set innodb_file_format=Barracuda and innodb_large_prefix=1 on mariadb on an-coord1001 to allow bigger indexes for Superset db upgrades [15:00:53] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to EventGate - https://phabricator.wikimedia.org/T211248 (10Ottomata) [15:00:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:00:55] ottomata: --^ [15:01:06] going to puppetize it (Manuel suggested the changes) [15:18:51] ok! [15:19:12] 10Analytics-Kanban, 10EventBus, 10Operations, 10netops: Allow analytics VLAN to reach schema.svc.$site.wmnet - https://phabricator.wikimedia.org/T221690 (10ayounsi) a:03ayounsi [15:25:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10elukey) New issue opened: https://github.com/apache/incubator-superset/issues/7368 [15:27:10] hey I can't make Scrum of Scrums, if someone wants to go for me, that'd be coo [15:27:12] *cool [15:31:22] nuria: https://github.com/apache/incubator-superset/issues/7368 - lol [15:34:51] Hi! Can I get added as a user to Superset? I'm already part of the wmf LDAP group but am currently getting "AttributeError: 'bool' object has no attribute 'login_count'" [15:35:15] Mneisler: sure! what is your ldap ? [15:37:22] mneisler [15:39:57] Mneisler: can you retry now? [15:41:18] I can log in now. Thanks @elukey! [15:46:38] 10Analytics, 10Analytics-Kanban: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608 (10Nuria) I see, +1 to naming then if this is some existing media wiki convention. [15:52:18] so it is interesting that now users not yet created in superset get the following [15:52:21] https://github.com/dpgaspar/Flask-AppBuilder/issues/432 [15:52:42] elukey: WE WILL NEVER MOVE from 0.31 is been made clear! [15:53:01] ottomata: this seems a different issue from the one that you fixed with Flask upstream right? [15:53:33] nuria: I was able to build and deploy 0.32 without our wikimedia branch, so if I get those things fixed it should be very easy to upgrade [15:53:35] elukey: i do not think it was ever fixed [15:53:54] IIRC the issue was too many redirects at the time though [16:00:51] ping fdans standdduppp [16:39:26] ERROR:flask_appbuilder.security.sqla.manager:Error adding new user to database. (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry '-' for key 'email'") [SQL: 'INSERT INTO ab_user (first_name, last_name, username, password, active, email, last_login [16:39:36] what a beauty [16:45:45] elukey: jajajaja [16:46:33] elukey: ya, that was the problem that andrew found [16:46:42] elukey: i mean funny but not really [16:55:17] 10Analytics, 10Discovery, 10EventBus, 10Wikidata, and 5 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731 (10Nuria) 05Open→03Resolved [17:05:08] nuria: ok I thought that we were waiting for upstream to merge something, but it seems not [17:05:30] in theory if the ldap auth worked with groups we could directly use it [17:09:56] or maybe removing the unique constraint in the table? [17:10:32] elukey: in what field is the constrain unique? [17:10:40] UNIQUE KEY `email` (`email`), [17:10:41] elukey: cannot be email....os is it? [17:10:48] elukey: WELL I WAS WRONG [17:10:55] there are two [17:10:56] UNIQUE KEY `username` (`username`), [17:10:56] UNIQUE KEY `email` (`email`), [17:11:01] the first is understandable [17:11:08] the latter not sure [17:11:15] elukey: wait [17:11:31] elukey: i think the constrain should be on the email value, makes sense [17:11:55] I am not sure, the username is already unique (it should be) [17:12:05] elukey: is this happening cause by default it is inserting blank email values? [17:12:19] yeah, since it gets only the remote user via httpd [17:12:25] elukey: ah ok [17:12:33] elukey: ya, +1 to remove unique constrain [17:12:36] so it uses '-' that is already used [17:12:53] elukey: ya, i thought it might be blank but ya [17:24:32] going off! [17:24:49] tomorrow is bank holiday in Italy (Liberation day), forgot to mention during standup [18:26:53] bbiab [19:11:57] (03PS1) 10Bmansurov: Oozie article recommender: use version 0.0.2 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/506218 (https://phabricator.wikimedia.org/T210844) [19:25:49] (03PS2) 10Bmansurov: Oozie article recommender: use version 0.0.2 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/506218 (https://phabricator.wikimedia.org/T210844) [19:53:37] elukey: liberation from what, if I may ask? [19:54:07] (oof, sorry for the ping at the late hour) [19:58:41] 10Analytics, 10Analytics-Kanban, 10EventBus, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10SBisson) Does it mean we can now have a fie... [20:49:24] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) @fsero @akosiaris - moving discussion about eventgate-main patch here,... [20:54:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) > Is the values.yaml file in the released chart used at all when relea... [21:35:35] 10Analytics, 10Analytics-Kanban, 10EventBus, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Ottomata) It almost does! We're blocked by... [21:38:36] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Maybe: - chart: eventgate -- namespaces: analytics, main, logging ---... [22:07:40] 10Analytics, 10Research, 10Wikidata: Copy Wikidata dumps to HDFs - https://phabricator.wikimedia.org/T209655 (10abian) @JAllemandou, do you think this is now unblocked? [22:33:39] 10Analytics, 10Research, 10Wikidata: Copy Wikidata dumps to HDFs - https://phabricator.wikimedia.org/T209655 (10Nuria) @abian : this is still not happening on a recurrent schedule yet. [22:34:50] hare: the facists, it is the bella ciao day: https://en.wikipedia.org/wiki/Bella_ciao#Partisan_version