[08:02:54] (03CR) 10Ladsgroup: [C: 032] "Let it begin." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292508 (owner: 10Ladsgroup) [08:03:50] (03Merged) 10jenkins-bot: Use ores.wikimedia.org instead of ores.wmflabs.org [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292508 (owner: 10Ladsgroup) [09:00:41] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2360251 (10schana) >>! In T134782#2353416, @yuvipanda wrote: > Depending on what you mean by 'internal-to-labs', the answer might be 'lol, we do not... [13:20:13] halfak: around ? [13:20:58] halfak: I was looking at https://www.mediawiki.org/wiki/Wikimedia_Research/Goals/RDFY17Q1 and was wondering what a DBA is needed for on the Focus goal [13:47:26] Hey akosiaris [13:47:37] ο/ [13:48:07] The last time we talked to Jaime about deploying the ORES extension, he warned us of some concerns over the DB use that the extension needs. [13:50:29] * halfak looks through phab but fails [13:51:35] Looks like this is the most relevant I can find: https://phabricator.wikimedia.org/T124443 [13:52:07] But there *was* a conversation with jaime where he said that he thought some of the slices couldn't handle the additional data needed by the extension. [13:52:12] Amir1, around? [13:52:25] I wonder if you can help find this past discussion [13:56:11] halfak: ok, that gives me enough context for now. I 'll ping Jaime to get some more info [13:56:24] but now it makes sese [13:56:26] sense* [13:56:43] :) Please loop me in when you ping him [13:56:49] I'd like to turn that N into a number [13:58:24] heh, not sure you will like the number but I will [14:03:08] No worries. I like reality. [14:07:33] halfak: so I think you were referring to https://phabricator.wikimedia.org/T123795#1973644 [14:07:35] correct ? [14:07:50] Yes! [14:07:51] \o/ [14:08:12] "I expect those to be here by 15 March." [14:08:19] Let's hope that estimate was right [14:08:22] Jaime's concerns on that one are no longer there. We got new hardware so we can probably move forward with everything [14:08:32] I 'll get Jaime to update that task [14:08:38] Great. We only support 20 wikis ATM [14:08:41] Thanks! [14:20:15] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2360919 (10jcrespo) Hi, most space issues have been corrected since the last time we talked due to the... [14:36:09] o/ schana [14:36:36] Any progress on refactor design since we last talked? [14:37:03] Your notes on https://phabricator.wikimedia.org/T137125 [14:37:10] Look bigger than the goals I had set forth [14:42:27] I wonder if it might make sense to finish the work that I started and leave this refactor for something bigger. [14:43:55] hey halfak [15:21:12] Hey schana [15:21:16] So any thoughts? [15:22:05] I don't want to hold up any progress [15:22:42] I think it might make sense to have a conversation about how feature extraction, injection, etc. work [15:22:49] Agreed [15:22:53] Since it seems that you want to go in that direction as a component [15:23:05] where would you like to maintain a longer-term, async conversation? [15:23:12] +1 [15:23:39] This larger context could let you question whether we want to stick with the distributed queue processing strategy too. [15:24:11] One option we haven't seriously explored is dropping the whole queue and going with stateless uwsgi workers [15:24:20] I think there is *a lot* of benefit to be had from a distributed setup with message passing [15:24:28] An independent feature extraction process would make that more tractable. [15:24:38] schana, a lot of complication too [15:24:49] well, that depends on the scope [15:25:12] Does it? Isn't complication still complicated when well scoped? [15:25:38] if the workers are all stateless and single-minded... [15:26:38] and celery is simply used as a retry-on-failure and message passing infrastructure connecting individual components [15:50:36] 10Revision-Scoring-As-A-Service-Backlog: [Spike] Could we build a "list article" quality model - https://phabricator.wikimedia.org/T136006#2361209 (10Halfak) Grades are: * FL - Featured list * GL - Good list There's nothing below that that we could use for training/testing. So I'm worried that it'll be hard t... [15:51:22] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2361212 (10jcrespo) @Ladsgroup, I have added you to the grafana-admin LDAP group, please check that you can log in correctly to https://grafan... [15:54:22] First worklog entry in a long time! https://meta.wikimedia.org/wiki/Research_talk:Automated_classification_of_article_quality/Work_log/2016-06-07 [16:13:00] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2361259 (10yuvipanda) Currently, there is no clear way to do that. [16:38:35] halfak: hey [16:38:48] o/ [16:38:51] I just woke up [16:38:55] what's up [16:39:11] akosiaris was asking about database needs for the ORES extension [16:39:15] We got it worked out though :) [16:39:24] Jaime told us that s2, s3 have storage issues [16:39:45] so we can't deploy the extension which takes a big amount of storage in those wikis [16:40:01] (s2, s3 contains most of wikis) [16:40:59] Looks like that has been resolved [16:41:07] As of March :) [16:41:26] Looks like Jaime wants updated size estimates now [16:41:52] but eta for fixing it was early April, so it should be fixed by now [16:42:44] Messages must be coming through slow [16:43:14] sure [16:43:21] I will get that asap [16:47:41] Cool [16:49:18] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2361363 (10Ladsgroup) Hey, Yes. I was able to login to the grafana-admin. Thanks [16:54:57] Building features for ruwiki :) [16:57:28] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2361455 (10jcrespo) 05Open>03Resolved [17:23:39] Looks like we're getting decent accuracy for the ruwiki classifier [17:25:32] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2361604 (10Ladsgroup) @jcrespo Hey, Worst case scenario, after three years: fawiki: 401 MB, 12 GB for en... [17:38:00] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2361659 (10Ladsgroup) Sure, that's the plan. Thanks :) [17:38:45] halfak: https://phabricator.wikimedia.org/T123795#2361652 [18:25:50] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2361789 (10Halfak) OK. Let's try out an example iterative plan. Let's say we do the iterations below w... [18:26:12] Amir1, https://phabricator.wikimedia.org/T123795#2361789 [18:28:55] halfak: I think after deployment of the first few wikis, things will change [18:29:17] What will change? [18:29:18] number of wikis with best support will go up [18:29:24] Oh yeah. I agree. [18:29:49] I think it's hard to plan for that future now [18:29:57] I like doing the ones that currently have less support last so that maybe they'll have better support by the time we get there. [18:30:05] I don't think we need to plan for it at all. [18:30:27] At most, they'd wait a week after wikis that currently *do* have good support. [18:32:08] yeah [18:32:33] but I think having the damaging model should be mandatory for those wikis [18:34:37] Really? I think it would be good to deploy before we have the damaging model ready and then hold the labeling campaign as an incentive for better accuracy. [18:35:49] the thing is that the extension actually can store scores for multiple models but it only works in RC when damaging is there [18:36:11] that was the decision that was made by the time I got there. [18:36:31] but I can imaging having it wouldn't be hard [18:37:51] Hmm... Seems like this could be important. [18:38:05] The system should be able to store an arbitrary set of scores for a single change. [18:38:36] it can and it does [18:38:47] Oh good. I must be missing the problem. [18:38:51] but it doesn't use them in the RC GUI [18:39:03] it only uses the damaging model [18:39:24] Hmm... that's a problem. [18:39:27] It should be configurable [18:40:36] implementing it isn't hard [19:15:21] halfak: one thing, it would be great to start experimenting with performance and stuff [19:15:35] so we can deploy the extension faster [19:15:56] Yeah. You are right. I'm just finishing up the ruwiki modeling stuff. [19:16:01] I'll pick that up next. [19:16:16] been looking at the ORES api client. I have some code for that. There will be a PR soon. [19:17:14] \o/ [19:33:49] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: implement an option in preferences to make hidenondamaging default - https://phabricator.wikimedia.org/T130560#2361984 (10Quiddity) [19:46:10] Oh, I almost forgot [19:46:30] halfak: is it okay to put the prod in the weekly update [19:46:31] ? [19:46:51] Yes. [19:46:59] But maybe note that we're still testing. [19:48:46] Okay [20:21:05] halfak: https://etherpad.wikimedia.org/p/ores_weekly_update [20:21:18] I add the link after your edits [20:21:20] :) [20:56:34] afk [20:56:47] be back in a half an hour [20:57:00] kk [20:57:02] PR coming in soon [21:14:49] https://github.com/wiki-ai/wikiclass/pull/23 [21:14:53] ^ ruwiki model [21:15:00] ORES api client incoming too [21:20:07] https://github.com/wiki-ai/ores/pull/146 [21:20:09] BAM [21:22:35] back [21:22:41] nice [21:22:53] * Amir1 impinges halfak dropping the mic [21:23:31] halfak: it would be great if you review the weekly update before I post it away [21:29:27] Oh yea! [21:29:29] * halfak clicks [21:34:48] halfak: one thing about grafana (I've got the access!) [21:34:58] we need to move the current ores dashboard to [21:35:00] ores-labs [21:35:08] Sure. It seems we can copy it. [21:35:09] make ores for our setup in prod [21:35:23] Let's do that all in one big batch when we make the switch [21:35:38] yup [21:35:45] * yuvipanda is happy to see the deployment finally get done [21:35:51] It seems it needs to happen asap [21:35:56] major kudos to Amir1 for pulling it all the way to the end [21:36:14] yuvipanda: thank you for your support and advice, it wasn't possible without you [21:36:21] I will have the flight in 48 hours [21:36:55] one other thing: the beta labs setup now uses the ores in prod and it works just fine [21:37:08] I mean the beta wikis [21:37:46] +1 woo Amir1 [21:38:15] Definitely deserves the majority of the credit ORES actually landing in prod [21:38:27] Second is akosiaris. [21:38:33] I'll take third, I think :) [21:39:28] Building something that *can* work in prod was the biggest obstacle, moving it to prod is just some settings [21:39:53] thus I give you, yuvipanda the biggest credit [21:43:49] halfak: thanks for adding the celery demo, I'm sorry for not adding it [21:44:00] I thought maybe, it's part of a bigger plan [21:44:45] It's OK. Just progress reporting. Maybe I'll add a link to the bigger refactor [21:47:57] sent [21:48:09] SPAM IS COMING [21:48:37] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: ORES extension should send a precache=true to the service when an edit is made - https://phabricator.wikimedia.org/T137007#2362421 (10Ladsgroup) 05Open>03Resolved [21:48:39] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Deprecate flower - https://phabricator.wikimedia.org/T137003#2362422 (10Ladsgroup) 05Open>03Resolved [21:48:40] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10revscoring: Score multiple models with the same cached dependencies - https://phabricator.wikimedia.org/T134606#2362424 (10Ladsgroup) [21:48:41] Come forth Phab tasks! [21:48:43] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2362423 (10Ladsgroup) 05Open>03Resolved [21:48:45] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review, 07Puppet: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2362425 (10Ladsgroup) 05Open>03Resolved [21:48:45] OMG [21:48:47] WOOO [21:48:47] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test `reverted` model for nowiki - https://phabricator.wikimedia.org/T131856#2362428 (10Ladsgroup) 05Open>03Resolved [21:48:49] 06Revision-Scoring-As-A-Service, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2362429 (10Ladsgroup) 05Open>03Resolved [21:48:51] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test `reverted` model for nowiki - https://phabricator.wikimedia.org/T131856#2180730 (10Ladsgroup) [21:48:59] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure: ores-beta is down - https://phabricator.wikimedia.org/T135677#2362427 (10Ladsgroup) [21:49:02] 06Revision-Scoring-As-A-Service, 03Scap3: Scap3 checks should not be random - https://phabricator.wikimedia.org/T136253#2362426 (10Ladsgroup) 05Open>03Resolved [21:49:04] 06Revision-Scoring-As-A-Service, 10Wikilabels: [Investigate] Intermittent performance issues with wikilabels - https://phabricator.wikimedia.org/T130872#2362431 (10Ladsgroup) 05Open>03Resolved [21:50:24] back to checking logs of beta cluster [21:58:17] OK. I'm off to go do some non-work-paperwork. Will be on gtalk and telegram. [21:58:19] o/ [21:58:56] o/ [22:16:37] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test `reverted` model for nowiki - https://phabricator.wikimedia.org/T131856#2362537 (10Ladsgroup) 05Open>03Resolved [22:25:01] yuvipanda: when you have about a min, It would be great to +2 this: https://gerrit.wikimedia.org/r/#/c/293225/ [22:25:22] super trivial [22:25:28] Amir1: can you verify that that package exists in debian jessie? [22:25:31] if so I can merge [22:25:44] let me check for sure [22:25:56] I did it in Ubuntu and it worked [22:26:07] right, but ubuntu != debian jessie :D [22:26:14] of course [22:26:17] wait a sec [22:26:23] we had this problem with the indonesian one It hink [22:26:29] where there was a package for ubuntu but not for jessie [22:26:35] I 'fixed' it by importing the package [22:27:03] https://packages.debian.org/jessie/myspell-nb [22:28:00] strange [22:28:30] why strange, Amir1? [22:28:53] that a spelling dictionary is not in jessie but in ubuntu [22:29:01] ah yeah [22:29:04] it was funky [22:29:08] Amir1: anyway, I'm going to merge now [22:29:13] \o/ [22:29:14] thanks [22:33:52] Amir1: I merged it btw (forgot to tell here) [22:34:04] no worries, thanks :)