[07:53:10] 10Scoring-platform-team, 10MediaWiki-JobQueue, 10ORES, 10Performance-Team, and 5 others: Job queue corruption after codfw switch over (Queue growth, duplicate runs) - https://phabricator.wikimedia.org/T163337#3406080 (10elukey) Today I tried to count the occurrences of `EVALSHA 3917a6b71d2857ca3d40fe6dabc2... [12:57:35] 10Scoring-platform-team-Backlog, 10ORES, 10Patch-For-Review: Implement twemproxy for ORES in production - https://phabricator.wikimedia.org/T122676#3407217 (10akosiaris) Change above has been merged, all that's left is the above is to turn the knob and enable it. That would be https://gerrit.wikimedia.org/r/... [13:06:20] o/ [14:08:51] o/ Amir1 [14:08:59] Saw your notes on the thresholds PR in revscoring [14:09:22] Not sure what you are hoping for re. "smaller" and the conflicts came due to your recent pep8 PRs [14:13:32] halfak: I mean things that are not related directly to the refactor gets done outside of the PR (in other PRs) [14:13:41] so It's easier to review [14:13:43] Like what? [14:14:43] halfak: 1- the change in .gitignore file [14:14:58] Even minor little things like that? You want a seperate PR? [14:15:05] 2- config/gradient_boost.params.yaml [14:15:09] and so on [14:15:24] the thing is these are piling up and makes the main thing impossible to properly review [14:16:05] Also it makes it more prone to errors like https://github.com/wiki-ai/revscoring/pull/307/files#diff-59f7df10a1018b844cc15aafbbc1945e [14:16:28] That's not really an error. [14:16:36] I can remove that file if it is confusing. [14:16:44] It helped me think while I was engineering. [14:17:13] no the "..." makes it invalid json [14:17:21] see my comment [14:17:38] Right. It's never supposed to be evaluated. your comment is just a "?" [14:18:46] I thought it's obvious that it's invalid (github flags the whole thing red) [14:20:43] Right, but is it not clear that it's not intended to be valid JSON? [14:20:51] Either, way, I've removed it. [14:21:28] If I were to removed the few minor cleanups I did from this PR, it would get 2% smaller. [14:23:03] The biggest problem is that github refuses to show a diff for a lot of files. [14:23:16] Instead, it shows the file as deleted and then created from scratch. [14:24:33] Yeah this is big bummer [14:24:51] I can blindly merge it but I don't want an outage [14:25:09] Maybe it would be helpful if you focused on the tests. [14:25:09] at least something I can look into properly [14:25:13] That way you can see how it all works. [14:26:20] okay, I give it another try later today [14:26:30] I need to be afk for half an hour [14:26:34] I'll be back [14:26:49] o/ [14:26:52] Thanks dude. [14:50:00] 10Scoring-platform-team-Backlog, 10ORES, 10Patch-For-Review: Implement twemproxy for ORES in production - https://phabricator.wikimedia.org/T122676#3407572 (10Halfak) Ok. This *looks* simple. Is tewmproxy known to be running on "localhost" and in a good state? When we flip the switch, will we be finding o... [14:50:37] akosiaris, ^ [14:50:43] If you want to discuss here. [16:18:38] Great! The draft quality w/ and w/o sentiment models finished training. [16:38:15] File "/home/awight/.env/lib/python3.4/site-packages/revscoring/scorer_models/scorer_model.py", line 73, in load [16:38:18] return pickle.load(f) [16:38:20] ImportError: No module named 'draftquality' [16:38:42] pickle problems. [16:39:36] Might not be in the right folder for python to find "draftquality" in scope [16:40:16] It's discoverable from a command-line python interpreter, though [16:40:35] What's PWD? [16:40:55] >>> import draftquality [16:40:56] >>> draftquality [16:40:56] [16:41:04] I'm in /srv/awight/draftquality [16:41:20] Oh i see [16:41:20] * halfak squits [16:41:26] Not sure what's up [16:41:34] draftquality comes from the cwd, while other modules come from virtualenv, that might be related. [16:41:42] I'll symlink the two... [16:43:51] haaargh [16:44:08] my with_cache observations are missing draftquality_with_sentiment [16:44:52] See anything wrong with, wikiclass extract_from_text \ [16:44:52] draftquality.feature_lists.enwiki.draft_quality \ [16:44:52] draftquality.feature_lists.enwiki.draft_quality_with_sentiment \ [16:44:52] --extractors 12 \ [16:45:01] That looks right to me [16:47:18] ah basic misunderstanding... [16:48:31] The feature lists are distinct, but I was also giving "draft_quality_with_sentiment" as the label. We actually want normal "draft_quality" [16:49:06] Aha! [16:49:38] k I think tests are running now. [16:49:52] Anyone have things I should report in scrum of scrums? [16:50:05] We're not blocked or blocking? [16:50:15] awight, just the stuff in the ether pad from me. [16:50:38] halfak: which etherpad? our Monday notes? [16:50:58] Yeah. We have a section in the etherpad. We talked about it in the meeting :D [16:51:18] :) looking [16:52:45] 10-4 [17:17:18] halfak: since it's too big to review, I suggest we merge it and then run extensive tests on it [17:17:43] Amir1, I'm OK with that. I'll need to do a lot more docs before we merge. [17:18:00] If awight is OK with that plan too, then I'll flesh out the docs and we can have one more discussion before merging. [17:18:52] I generally lean that way, it just turns into bookkeeping when you have to maintain a giant patch. [17:18:55] however... [17:19:12] halfak: okay, I also fixed the GUI patch, the comment you made [17:19:28] We need to figure something out to avoid humongous patchball in the future [17:19:50] Amir1, just added another content. Rendering is still messy. [17:19:55] With gerrit, it's simpler, you can just cut the change up into smaller patches and merge them individually. [17:19:59] awight, hard with refactorings. [17:20:14] I'm not sure how to do it with GH, cos feature branches don't stack on top of one another nicely [17:20:58] Much easier for more minor changes and other purely new code. [17:21:02] halfak: I see that sometimes it's less effort to just commit all of the things that you want to change, when they're interdependent and there's no reason to do actual migrations between each step [17:21:09] but e.g. this commit https://github.com/wiki-ai/revscoring/pull/307/commits/9a10b25f18d2bd428258feb2539ca5005dcc655f?diff=unified [17:21:16] could have gone in by itself [17:21:31] not that I'm lobbying for this specifically, but just as an example. [17:21:51] Hmm... GOtcha. So a commit to add the class but leave it totally unused? [17:22:02] I prefer that personally. [17:22:17] Interesting. I think a totally unused class would be cause to reject a PR [17:22:20] But it's a collective style decision, I guess [17:22:24] hm [17:22:30] Maybe just haven't thought this way before [17:22:58] well I usually like to add a bunch of functionality, discuss it in isolation and get it set up where it needs to go, then commit a one-liner that switches over to the new thing. [17:23:21] not that I'm very good at it... [17:26:49] I can also appreciate that it might require more cognitive overhead to merge such discrete patches that it's hard to go back and reference why supporting changes were made. [17:27:03] So I'm fine with whatever your working style is at the time. [17:27:21] Just don't make me read 3k lines twice :p [17:27:47] Interesting. I'd be interested in giving that a try next time [17:27:53] I can sort of imagine how that would go. [17:28:12] FWIW, the changes is really more like 1.5k lines ;0 [17:28:20] Stupid github/git [17:28:51] halfak: ^ It's a little bit better but I need to rewrite the whole thing later on [17:29:26] halfak: in that case, I'll have some comments on the medium-sized patchball soon :) [17:30:00] Amir1, looks good :) [17:30:23] thanks awight. [17:34:29] Everyone have a good 4th of july? [17:34:43] o/ [17:34:54] Yeah. Saw some 'splosions. Worked in my yard :) [17:35:07] Love me good splosions [17:35:43] Also hows the server change to our own cluster going aaron? [17:35:52] No updates from last discussion [17:36:00] akosiaris may know better than me. [17:36:19] I'm off to lunch [17:36:22] Back in a bit. [17:36:27] Ok see u in a bi [17:36:28] Bit [18:11:07] halfak|Lunch: Now I get what you were saying about this diff--git/github didn't follow the renames :( [18:11:24] :/ [18:29:52] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, and 2 others: [Discuss] Make ORES Review Tool preferences more prominent - https://phabricator.wikimedia.org/T167910#3349272 (10ggellerman) @halfak @Ladsgroup Should w... [18:32:06] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, and 2 others: [Discuss] Make ORES Review Tool preferences more prominent - https://phabricator.wikimedia.org/T167910#3408578 (10jmatazzoni) I want to review this as well... [18:35:08] halfak: Did you say, "microgradient" of the ROC-AUC? https://en.wikipedia.org/w/index.php?title=Microgradient&action=edit&redlink=1 [18:35:18] awight, right! it's crazy. I've done everything that I can think of. [18:35:33] microgradient? [18:35:41] ah okay I misheard [18:35:46] Did I say that somewhere? [18:35:57] you had some new fitness metric [18:36:10] Could be the "Area under the Receiver operation characteristic" == ROC-AUC [18:36:24] I thought I saw the new metric fly by [18:36:25] AUC = Area under the Curve [18:36:30] among your patchblitz :p [18:36:40] stats[roc_auc][micro] [18:38:41] Oh yeah. That's a micro-average [18:39:13] http://rushdishams.blogspot.com/2011/08/micro-and-macro-average-of-precision.html [18:40:30] ty [18:41:50] weird. [18:42:31] Ack. Looks like some crap stuck around in an old directory [18:42:39] Cleaning that up now. Must have been from a rebase. [18:44:03] Wait. No [18:44:12] I just think I'm seeing comments on a very old commit. [18:44:16] hehe, we'd better get this merged before you implode [18:44:36] Saw some comments from you on that old "Thresholds" class. [18:45:00] Which has long since been removed. [18:45:10] ack. I did in fact do what you suspected. [18:45:23] well, I'm reviewing the entire change now [18:54:27] O/ [18:54:36] So! I'm going to look into the regex timing now [18:54:44] I know codezee was thinking of doing that. [18:54:50] But I think it's about time I took a look [18:56:12] godspeed [19:23:09] fud. [19:23:31] food* [19:25:15] s/f[uo]+d//g [19:26:00] to eat it all [19:26:03] "?" non-greedily [19:52:41] wiki-ai/revscoring#1093 (master - c60b332 : Amir Sarabadani): The build was broken. https://travis-ci.org/wiki-ai/revscoring/builds/250495773 [20:05:41] wiki-ai/revscoring#1096 (master - 33ec3c6 : Amir Sarabadani): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/250501064 [20:33:25] 10Scoring-platform-team, 10WMF-Communications, 10Wikimedia-Blog-Content: Announce new team: "Scoring Platform" - https://phabricator.wikimedia.org/T169755#3409223 (10Halfak) [20:39:22] Woops. Looks like that thresholds PR got merged. I'll need to do some followups. awight had some good notes too. [20:39:35] I'll make sure to find time to go apply those notes in a followup tomorrow. [20:43:18] All my comments were probably in response to preexisting conditions... [20:43:22] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Set up larger ores-compute instance - https://phabricator.wikimedia.org/T169809#3409254 (10Halfak) [20:45:27] I'm curious, does wmflabs put VMs like ores-compute on dedicated hardware? Are we using virtual cores or real cores? [20:48:21] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Increase quote for ores-staging to 48 GB - https://phabricator.wikimedia.org/T169811#3409290 (10Halfak) [20:49:09] halfak: oh! another patchblob strategy, something AndyRussG showed me. You file mega-PRs like a refactor against a new branch, rather than master. Then we can merge your PRs independently of whether there are still WIPs or tests don't pass, and catch followup stuff before landing on master. [20:49:23] In theory, it's even compatible with github workflows [20:50:27] awight, oh. I like that idea. [20:50:46] It lets us have another level of granularity. [20:52:27] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Set up larger ores-compute instance - https://phabricator.wikimedia.org/T169809#3409318 (10Halfak) [20:52:46] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Set up larger ores-compute instance - https://phabricator.wikimedia.org/T169809#3409254 (10Halfak) [21:19:07] 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: Use our own scoring models in `tune` utility - https://phabricator.wikimedia.org/T163711#3206783 (10awight) 05Open>03Resolved [21:19:50] 10Scoring-platform-team, 10revscoring, 10Easy, 10artificial-intelligence: Store the detailed system information inside of model files. - https://phabricator.wikimedia.org/T160223#3092590 (10awight) PR #307 resolves this? [21:24:40] Amir1: fyi, I moved T149117 to "done" based on a comment saying the patch was manually merged... [21:24:41] T149117: ORES UI is broken - https://phabricator.wikimedia.org/T149117 [21:26:31] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Increase quota for ores-staging to 48 GB - https://phabricator.wikimedia.org/T169811#3409441 (10Halfak) [21:26:43] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Increase quota for ores-staging to 48 GB - https://phabricator.wikimedia.org/T169811#3409290 (10Halfak) [21:26:50] Hey. Shouldn't test_model be multithreaded? Same with testing inside of cv_train. [21:26:53] halfak: ^ [21:27:10] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Request increase quota for ores-staging to 48GB RAM - https://phabricator.wikimedia.org/T169811#3409290 (10Halfak) [21:27:36] ah, I guess cv_train is already using one core per model, so there's nothing to change there. [21:28:27] 10Scoring-platform-team, 10Cloud-VPS, 10ORES: Request increase quota for ores-staging to 48GB RAM - https://phabricator.wikimedia.org/T169811#3409290 (10Halfak) [21:29:41] and I'm a freak for using train_model. [21:32:32] 10Scoring-platform-team, 10Patch-For-Review: ORES puppet error on labs boxes, unable to set user to "deploy-service" - https://phabricator.wikimedia.org/T169164#3389235 (10awight) a:03awight [21:33:04] 10Scoring-platform-team-Backlog: Bury horrors of the editquality makefile - https://phabricator.wikimedia.org/T168455#3409477 (10awight) [21:41:41] awight are you looking into the puppet failures? [21:44:37] paladox: It's all yours if you want! I saw that akosiaris left a suggestion in gerrit, to switch the chown'd user depending on environment. [21:44:40] https://gerrit.wikimedia.org/r/#/c/362097/ [21:45:02] yeh, though what user can we use? [21:46:14] nobody/nogroup was a reasonable idea [21:47:02] I have to run away for 1-2hr, but see you tomorrow! [22:01:41] re. test_model, yes! It could be multithreaded! GOod idea. [22:05:22] halfak: what do you think of this? ^ [22:05:53] 10Scoring-platform-team, 10ORES, 10User-Ladsgroup: Apply mediawiki core styling convention on javascript files of ores - https://phabricator.wikimedia.org/T169577#3409586 (10Ladsgroup) https://github.com/wiki-ai/ores/pull/214 [22:06:23] I hate Javascript variable patterns. [22:06:24] do we have an actual deployment schedule or is it just whenever deployers want [22:06:26] :) [22:06:45] I hate standards like pep8 and eslint but im still here xD [22:07:32] Zppix, no schedule [22:07:49] wouldnt it be a good idea to setup one? [22:07:50] Amir1, I think it's pretty good. How do you run your eslinter locally? [22:07:54] Could you add that to come docs? [22:08:01] Zppix, we don't have the resources. [22:08:13] true... [22:09:23] halfak: yeah, you need to do npm install eslint [22:09:26] than eslint . [22:09:43] Oh cool. That's easy enough [22:09:49] I tried several ways honestly (like grunt and stuff) but this one was the easiest [22:10:19] will travis run tests? [22:12:45] Zppix: halfak I'm adding it to travis [22:12:54] cool [22:13:18] I really hope it doesn't make strange noises [22:15:18] if it does ill see if i can find why [22:17:11] https://travis-ci.org/wiki-ai/ores/builds/250546249?utm_source=github_status&utm_medium=notification [22:17:17] it passesssssss [22:17:18] yay [23:07:24] 10Scoring-platform-team, 10ORES, 10revscoring, 10artificial-intelligence: Why don't timeouts work during long regular expression matching? - https://phabricator.wikimedia.org/T168965#3409787 (10Halfak) https://gist.github.com/halfak/b31b8ddc38ca701c4c964478a53da75f I can't get the degenerate behavior in t... [23:27:06] 10Scoring-platform-team, 10ORES, 10Cloud-VPS (Quota-requests): Request increase quota for ores-staging to 48GB RAM - https://phabricator.wikimedia.org/T169811#3409928 (10bd808)