[10:44:48] 10Revision-Scoring-As-A-Service-Backlog, 10Bad-Words-Detection-System, 10revscoring: Add language support for Meta-Wiki - https://phabricator.wikimedia.org/T144121#2590378 (10Ladsgroup) @MarcoAurelio since metawiki is multi-lingual, I guess we can skip this part. But one question. What re the most commonly u... [10:47:13] 10Revision-Scoring-As-A-Service-Backlog, 10Bad-Words-Detection-System, 10revscoring: Add language support for Meta-Wiki - https://phabricator.wikimedia.org/T144121#2590381 (10MarcoAurelio) >>! In T144121#2590378, @Ladsgroup wrote: > @MarcoAurelio since metawiki is multi-lingual, I guess we can skip this part... [10:50:18] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train reverted model for metawiki - https://phabricator.wikimedia.org/T144163#2590382 (10Ladsgroup) [10:50:36] 10Revision-Scoring-As-A-Service-Backlog, 10Bad-Words-Detection-System, 10revscoring: Add language support for Meta-Wiki - https://phabricator.wikimedia.org/T144121#2589162 (10Ladsgroup) 05Open>03Invalid [10:51:00] 10Revision-Scoring-As-A-Service-Backlog, 10Bad-Words-Detection-System, 10revscoring: Add language support for Meta-Wiki - https://phabricator.wikimedia.org/T144121#2589162 (10Ladsgroup) It will be done soon [13:02:57] o/ [13:08:34] hey halfak [13:08:47] I'm deploying the enwiki change in the swat [13:09:10] Which change is this? [13:11:34] disabling the enwiki other models [13:12:17] Gotcha. Cool [13:12:25] Which will calm down resource usage. [13:12:36] But what about the ORES deployment of new models and the maintenance scripts? [13:20:58] Hey Amir1. Sorry. I had to switch to mobile quick [13:21:32] halfak|Mobile: it will happen as we planned [13:21:34] :) [13:27:19] OK Back at my desk [13:31:38] I'm afk for coffee [13:31:46] be back soon [13:31:49] :) [14:13:22] So, it looks like my new model building got interrupted by an issue with ores-compute-01 [14:13:35] It's now been successfully rebooted and is churning along again. [14:18:24] OK. Now back to work on hash vectors [14:18:33] Looks like we might have some massive demand for PCFGs soon [14:38:26] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train reverted model for metawiki - https://phabricator.wikimedia.org/T144186#2591095 (10Halfak) [14:40:45] o/ [14:40:52] Amir1: could you please review https://gerrit.wikimedia.org/r/#/c/264415/ again? [14:41:00] It's working fine locally [14:45:00] bmansurov: hey, sure. After the deployment but I can't +2 (I don't have rights) [14:45:31] Amir1: great, thanks. I'll +2 myself ;) [14:45:33] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train reverted model for metawiki - https://phabricator.wikimedia.org/T144163#2591120 (10Ladsgroup) [14:45:51] halfak: https://phabricator.wikimedia.org/T144163 [14:46:14] Woops. Nice [14:46:56] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train reverted model for metawiki - https://phabricator.wikimedia.org/T144163#2591123 (10Halfak) [15:00:40] o/ Amir1 [15:00:43] halfak: We are about to start the deployment, [15:00:47] Cool [15:00:52] I need ot grab something before we start [15:00:53] but [15:01:01] Have you finished all models? [15:02:05] I think we should deploy what is on beta now. [15:02:17] And worry about any model improvements for future deploys. [15:03:51] okay [15:03:56] the models are in beta? [15:04:14] Yeah. We did a deployment there a while back. [15:04:26] For you to test the maint script [15:04:36] awesome [15:04:40] https://ores-beta.wmflabs.org/v2/scores/enwiki/damaging/ [15:04:44] Confirmed "0.1.2" [15:04:54] Am working in tin right now [15:05:05] Current commit is 7aad8e9 [15:05:44] okay [15:05:48] Fetching submodules [15:05:51] (new models) [15:06:10] Amir1, should I be giving updates in -operations? [15:06:19] yeah [15:06:25] let's switch there [15:46:49] 06Revision-Scoring-As-A-Service, 10ORES: Set max-age header to 0 seconds for ORES to quiet secondary caches - https://phabricator.wikimedia.org/T144193#2591319 (10Halfak) [15:55:52] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Check model version replaces every time it runs. - https://phabricator.wikimedia.org/T144195#2591389 (10Ladsgroup) [15:58:13] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Get model version needs to invalidate cache. - https://phabricator.wikimedia.org/T144196#2591408 (10Ladsgroup) [16:23:48] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Implement ORES service proxy in api.php - https://phabricator.wikimedia.org/T143895#2591680 (10Anomie) >>! In T143895#2588608, @Ladsgroup wrote: > Yeah, I thought about that but it's not very nice saying the user... [16:47:30] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: [Discuss] api.php integration with ORES - https://phabricator.wikimedia.org/T122689#2591857 (10Halfak) [16:51:34] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Deploy ORES at 2016-08-29 - https://phabricator.wikimedia.org/T144101#2588574 (10Halfak) a:03Halfak [16:54:38] o/. On mobile for a bit [16:54:47] Amir1: ^ [16:55:00] okay, I need to travel [16:55:04] some parts are done [16:55:07] I continue in home [17:18:01] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Purge model score should clean when there is no row is ores_model too - https://phabricator.wikimedia.org/T144216#2591992 (10Ladsgroup) [17:45:08] halfak: Are you around with this nick or halfak|Mobile ? [17:45:17] around again [17:45:48] the populate database script is done for wikidata, fa, en, pl, nl [17:46:00] pt is running [17:47:57] nice [18:03:14] halfak: I was thinking if we should report an incident or not. The backend was mess but ores review tool worked just find for users (when we haven't updated the ores_model table, it scores them with the new model but stores it under old model, which I need to fix that later) [18:03:49] the maintenance script is done now [18:03:54] I don't think we should report it. [18:04:01] It wasn't really an incident. [18:04:30] It was not even really a meaningfully degraded service quality. [18:05:40] (03PS1) 10Ladsgroup: Do not clean based on oresc_rev, do it based on oresc_id [extensions/ORES] - 10https://gerrit.wikimedia.org/r/307345 [18:05:52] ^ when I saw it I was like whyy??? [18:20:34] (03PS2) 10Ladsgroup: Improvements to purging cache: [extensions/ORES] - 10https://gerrit.wikimedia.org/r/307345 (https://phabricator.wikimedia.org/T144216) [19:03:04] (03CR) 10Ladsgroup: [C: 032] Improvements to purging cache: [extensions/ORES] - 10https://gerrit.wikimedia.org/r/307345 (https://phabricator.wikimedia.org/T144216) (owner: 10Ladsgroup) [19:04:09] (03Merged) 10jenkins-bot: Improvements to purging cache: [extensions/ORES] - 10https://gerrit.wikimedia.org/r/307345 (https://phabricator.wikimedia.org/T144216) (owner: 10Ladsgroup) [19:07:03] halfak: I guess we have a problem [19:07:15] https://www.mediawiki.org/wiki/Topic:Takkdfuhsrxyti5b [19:08:19] The fastest way to fix it is to deploy this change https://gerrit.wikimedia.org/r/307345 and run maintenance script [19:23:27] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Redandat results in ORES review tool - https://phabricator.wikimedia.org/T144233#2592543 (10Ladsgroup) [20:31:00] Amir1, yikes! That's our fault? [20:31:10] some parts I guess [20:31:15] but I'm taking care of it [20:59:18] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Redundant results in ORES review tool - https://phabricator.wikimedia.org/T144233#2592797 (10Ladsgroup) [20:59:36] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Redundant results in ORES review tool - https://phabricator.wikimedia.org/T144233#2592543 (10Ladsgroup) enwiki, nlwiki, and fawiki are clean now. Working on wikidata and other wikis [21:09:26] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Redundant results in ORES review tool - https://phabricator.wikimedia.org/T144233#2592839 (10Ladsgroup) All wikis are clean now. Then I need to figure it out why it shows duplicate rows. [21:09:35] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Redundant results in ORES review tool - https://phabricator.wikimedia.org/T144233#2592840 (10Ladsgroup) p:05Unbreak!>03High [21:17:30] halfak: It's fixed now [21:17:45] Deployed? [21:17:47] I need to fix the underlying issue but It won't happen again :) [21:17:52] Nice! [21:17:57] I fixed it in-site [21:17:59] :D [21:18:28] directly cleaned ores_classification using a maintenance script called eval.php [21:19:16] because the maintenance script had a big issue, I need to backport the fix [21:19:39] also I need to make a patch for changing threshold [21:19:48] and write the weekly update [21:20:55] I get some rest and then I continue :) [21:25:21] I'll do the update [21:28:47] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Implement ORES service proxy in api.php - https://phabricator.wikimedia.org/T143895#2592920 (10Ladsgroup) If it's okay for you, it's okay for me. [22:06:18] I think I've addressed your notes akosiaris. Can you take another look? [22:06:22] https://github.com/wiki-ai/ores/pull/165 [22:06:44] (03PS1) 10Ladsgroup: Improvements to purging cache: [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/307438 (https://phabricator.wikimedia.org/T144216) [22:12:15] halfak: I backported and made the config change, I just need to wait for the deployment window (in 50 minutes) [22:36:45] Amir1, is the config change re. multiple rows? [22:36:53] Oh thresholds? [22:37:01] Amir1, please review https://etherpad.wikimedia.org/p/ores_weekly_update [22:46:14] halfak: thresholds [22:49:39] halfak: I made one small adjustment and I put a note [22:49:49] Ready to post then? [22:50:22] just change the OK to something else and watch out for typos too :D [22:54:09] awesome [22:55:09] OK. Sent :) [22:55:18] YESSS [22:55:21] SPAM COMING [22:55:33] :D [22:55:40] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup, 05WMF-deploy-2016-08-30_(1.28.0-wmf.17): Purge model score should clean when there is no row is ores_model too - https://phabricator.wikimedia.org/T144216#2593233 (10Halfak) 05Open>03Resolved [22:55:48] \o/ [22:55:53] * halfak swims on the SPAM FLOW [22:55:55] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 13Patch-For-Review: Deploy ORES at 2016-08-29 - https://phabricator.wikimedia.org/T144101#2593234 (10Halfak) 05Open>03Resolved [22:55:57] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup, and 2 others: Update model versions is badly broken in ORES extension - https://phabricator.wikimedia.org/T143798#2593236 (10Halfak) 05Open>03Resolved [22:56:06] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup: $wgOresModels set all models true - https://phabricator.wikimedia.org/T143988#2593235 (10Halfak) 05Open>03Resolved [22:56:08] \o_ [22:56:12] >--|o [22:56:13] 06Revision-Scoring-As-A-Service, 10Wikilabels, 10rsaas-editquality: Deploy 2016 edit quality campaign to English Wikipedia - https://phabricator.wikimedia.org/T143745#2593237 (10Halfak) 05Open>03Resolved [22:56:15] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2593238 (10Halfak) 05Open>03Resolved [22:56:23] 06Revision-Scoring-As-A-Service, 10rsaas-editquality, 15User-Ladsgroup: Include specific user groups in the trwiki edit quality model - https://phabricator.wikimedia.org/T140474#2593239 (10Halfak) 05Open>03Resolved [22:56:31] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 07Epic, 03Research-and-Data-2017-Q1: [Epic] Deploy ORES review tool - https://phabricator.wikimedia.org/T140002#2593240 (10Halfak) 05Open>03Resolved a:03Halfak [22:56:33] 06Revision-Scoring-As-A-Service, 10Wikilabels, 15User-Ladsgroup: Wikilabels UI reports non-200 status errors badly - https://phabricator.wikimedia.org/T138255#2593242 (10Halfak) 05Open>03Resolved [22:56:48] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: [Discuss] api.php integration with ORES - https://phabricator.wikimedia.org/T122689#2593243 (10Halfak) 05Open>03Resolved [22:57:25] BTW, Amir1, I've been working on the hash vector stuff. I'm almost done with the demo, but regretfully, it takes a very long time to generate my TFiDF training set. [22:57:27] See http://paws-public.wmflabs.org/paws-public/User:EpochFail/projects/vectors_demo/damage_detection_test.ipynb [22:57:51] nice! [22:58:08] But the fun story is that defining the whole pipeline to a feature vector is 10 lines! [22:58:24] I love tf-idf, they helped to discover curse words across all languages [22:58:41] AWESOME [22:58:42] I think that this pattern may, essentially, duplicate your work [22:58:57] But maybe in a more robust way. Also, maybe in a way that adds a different kind of signal. [22:59:28] Since it will catch 1grams, 2grams, 3grams, skipgrams, 12skipgrams, 13skipgrams, etc. [22:59:38] since our bad words and informal words got reviewed by human, I guess that's a signal [22:59:50] Agreed. The review helps us to not overfit [23:00:02] This strategy might result in overfitting [23:00:32] Bit will catch pattens like "his is a " [23:00:39] *patterns [23:01:06] I guess we might need more data [23:01:10] but not sure [23:01:26] Amir1, agreed. I think we should train the TFiDF on a big set of reverted [23:01:50] we might end up using self-training D: [23:01:52] *:D [23:02:17] Indeed [23:02:28] I'm going to take the dog to the park. I'll be back in 45 mins [23:03:24] halfak: https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=scb1001.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=mem_report&c=Service+Cluster+B+eqiad [23:03:33] we had 2 GB drop in memory usage [23:03:51] have fun [23:06:33] (03CR) 10Dereckson: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/307438 (https://phabricator.wikimedia.org/T144216) (owner: 10Ladsgroup) [23:14:46] (03Merged) 10jenkins-bot: Improvements to purging cache: [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/307438 (https://phabricator.wikimedia.org/T144216) (owner: 10Ladsgroup) [23:56:04] I'm head to bed [23:56:09] see you tomorrow [23:56:10] o/