[06:27:55] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.020 second response time [06:53:47] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 0.178 second response time [11:47:24] o/ [12:04:26] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Newsletter, and 5 others: Jenkins jobs for npm-test fail on project with deps on node-gyp which requires python2.7 - https://phabricator.wikimedia.org/T213944 (10zeljkofilipin) >>! In T2139... [12:32:08] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: Write a maintenance script to populate articlequality data for articles - https://phabricator.wikimedia.org/T194300 (10Ladsgroup) a:05Ladsgroup→03None [12:32:10] 10ORES, 10Scoring-platform-team: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Ladsgroup) a:05Ladsgroup→03None [12:32:12] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Malayalam-Sites, 10artificial-intelligence: Add language support for Malayalam - https://phabricator.wikimedia.org/T173193 (10Ladsgroup) a:05Ladsgroup→03None [12:32:14] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train a `reverted` model for jawiki - https://phabricator.wikimedia.org/T130869 (10Ladsgroup) a:05Ladsgroup→03None [12:32:19] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: PopulateDatabase should support time span and revids too - https://phabricator.wikimedia.org/T126461 (10Ladsgroup) a:05Ladsgroup→03None [12:32:28] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Spike] NLP for article quality models. - https://phabricator.wikimedia.org/T132533 (10Ladsgroup) a:05Ladsgroup→03None [12:32:37] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: [Spec] Use `reverted` models in ORES review tool - https://phabricator.wikimedia.org/T146378 (10Ladsgroup) a:05Ladsgroup→03None [12:33:49] 10ORES, 10Scoring-platform-team (Current), 10Performance, 10User-Ladsgroup: Make celery queues transient - https://phabricator.wikimedia.org/T210584 (10Ladsgroup) 05Open→03Declined [13:20:05] 10ORES, 10Scoring-platform-team: Build a test setup for redis sentinel in cloud VPS - https://phabricator.wikimedia.org/T210577 (10Ladsgroup) a:05Ladsgroup→03None [13:20:52] 10Scoring-platform-team, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737 (10Ladsgroup) [13:53:26] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10MW-1.32-notes (WMF-deploy-2018-05-01 (1.32.0-wmf.2)), 10Patch-For-Review, 10User-Ladsgroup: Store wp10 predictions in the MediaWiki database. - https://phabricator.wikimedia.org/T175757 (10Halfak) [13:53:29] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: Write a maintenance script to populate articlequality data for articles - https://phabricator.wikimedia.org/T194300 (10Halfak) 05Open→03Resolved a:03Halfak Looks like this is already done then. [13:54:35] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train a `reverted` model for jawiki - https://phabricator.wikimedia.org/T130869 (10Halfak) 05Open→03Resolved a:03Halfak Looks like this eventually got deployed. I can see it up on ORES. [14:22:53] afk for lunch [14:42:36] 10Scoring-platform-team, 10Wikidata, 10Wikidata-Campsite, 10editquality-modeling, and 2 others: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737 (10matej_suchanek) [14:53:32] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 3 others: Introduce a new namespace for collaborative judgements about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > How does it relate to the subset of wikitext used for edit summaries? As I understand you can do... [16:33:56] O/ [16:34:41] Traveling today. Was packing all morning. Am in the airport for the next couple of hours and then I'll be offline (flying) for the rest of the work day [16:35:14] I'll be in the wmf office around 4:30 PST to do some meetings FYI [16:46:37] dogspeed! [16:47:21] Thanks 😊 [19:14:23] ebernhardson: I thought you might be interested to know, we're doing work to get ORES scores and features into Hive... [19:14:58] The first use case will be to produce dump files, but I'm very interested in the potential for Spark ML once the data is available. [19:25:47] awight: nice. Spark ML is interesting, but you need pretty large datasets to make it worthwhile [19:27:10] i wonder what a feature store would look like, it's a bit early but at some point we likely want a semi-generic way to pull in "features about pages", like ores scores or whatever to feed into some process [19:27:51] ebernhardson: Well, one wrinkle we noticed recently is that we can only score revisions using the latest version of a model. For analytics and forensic purposes, it might be useful to score all records using all old model versions. That seemed like a good motivation to run the models in Hadoop. [19:28:13] ebernhardson: My current thoughts about feature storage are https://phabricator.wikimedia.org/T211069 [19:28:15] awight: right, but in that case you would run the models in hadoop using normal python code, not spark ML [19:28:59] ebernhardson: ah! Thanks for the learnin', that would be much simpler for us. [19:29:47] awight: i've found most hadoop things are simpler if instead of trying to do some distributed algo like spark ml, if you can simply split the work into 200 or 1000 independent pieces and run normal python code over it [19:30:33] That would work perfectly. [19:32:18] I'm completely ignorant of the performance concerns, but it almost looks like we could turn our pipeline around, extracting features directly into Hive and keeping up-to-date scores there, then serving from a lightweight cache. [19:38:16] harej: Hey, can I math something using your spreadsheet of the various wikis where ORES is deployed? [19:38:21] I want to sum the revision counts. [19:38:54] Go ahead [19:39:06] darn, it's not in that sheet. [20:11:45] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation, and 3 others: Decide whether we will include raw features - https://phabricator.wikimedia.org/T211069 (10awight) Just for fun, I elaborated on the quick estimate based on existing `w_cache` files. Note that these are not the "root" data sou... [20:17:39] 10ORES, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: Modify revscoring extract utility to include root datasources - https://phabricator.wikimedia.org/T214723 (10awight) [20:21:59] (03CR) 10Krinkle: ext.ores.highlighter: Fix minor memory alloc (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/485139 (owner: 10Krinkle) [20:25:30] (03CR) 10Krinkle: ext.ores.highlighter: Fix minor memory alloc (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/485139 (owner: 10Krinkle) [21:20:05] * awight curses at generators [22:20:10] wikimedia/editquality#456 (arg_typo - f0fc919 : Adam Wight): The build passed. https://travis-ci.org/wikimedia/editquality/builds/484569695 [22:46:40] o/ awight been awhile [22:52:50] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/486651 (owner: 10L10n-bot) [23:06:23] Zppix: howdy! What've you been up to lately? [23:07:46] awight: well i was busy studying for network certification and now i hold a CCENT [23:08:27] congratulations! That sounds like guaranteed employment. [23:12:25] 10ORES, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: Modify revscoring extract utility to include root datasources - https://phabricator.wikimedia.org/T214723 (10awight) WIP: https://github.com/wikimedia/revscoring/compare/roots?expand=1 [23:13:07] awight: Nothing yet :P been certified since oct [23:15:53] wikimedia/revscoring#1585 (roots - 275aca0 : Adam Wight): The build passed. https://travis-ci.org/wikimedia/revscoring/builds/484587420 [23:59:06] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues