[06:30:35] PROBLEM - check load on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:31:05] PROBLEM - check users on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:31:54] PROBLEM - check disk on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:32:23] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:52:35] RECOVERY - check load on ORES-web01.Experimental is OK: OK - load average: 0.52, 0.31, 0.63 [06:53:05] RECOVERY - check users on ORES-web01.Experimental is OK: USERS OK - 1 users currently logged in [06:53:54] RECOVERY - check disk on ORES-web01.Experimental is OK: DISK OK [06:56:48] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:36:54] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Upgrade numpy, scipy, sklearn - https://phabricator.wikimedia.org/T227023 (10Urstrulykkr) Done with the patch, https://github.com/wikimedia/revscoring/pull/442 waiting for your kind review. [11:32:11] 10Jade, 10Scoring-platform-team (Current), 10Design, 10User-Harej: Jade Wireframes: Entity edit mode - https://phabricator.wikimedia.org/T212374 (10Harej) a:05Harej→03None [11:38:28] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity view mode - https://phabricator.wikimedia.org/T212379 (10Harej) a:05Harej→03None [11:42:22] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity edit mode - https://phabricator.wikimedia.org/T212374 (10Harej) [11:49:49] 10Jade, 10Scoring-platform-team (Current), 10User-Testing, 10Design: Jade Wireframes: Diff view controls - https://phabricator.wikimedia.org/T212377 (10Harej) a:05Harej→03None [13:41:23] o/ [14:02:55] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:52:39] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:43:51] (03PS1) 10DannyS712: Add help links to special pages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/520470 (https://phabricator.wikimedia.org/T226938) [15:49:48] (03PS2) 10DannyS712: Add help links to special pages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/520470 (https://phabricator.wikimedia.org/T226938) [15:55:38] (03CR) 10jerkins-bot: [V: 04-1] Add help links to special pages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/520470 (https://phabricator.wikimedia.org/T226938) (owner: 10DannyS712) [16:23:36] accraze, I should make this clear: if we don't get the revert model together in time today, that is 100% OK. Let's just give it a good effort and see where we get :) [16:23:57] ok sounds good! [17:07:49] I'm going to go grab lunch. Back in 60 mins. [18:12:34] looks like no BS call today. That's OK with me. [18:15:05] haha oops! i totally spaced it 😬 [18:16:31] No worries if you're busy on something else [18:16:48] yeah almost done w/ tuning! [18:29:24] "BS"? [18:29:44] that's... remarkably frank, or it means something else :-D [18:30:08] apergos, yeah. It's a meeting with no work goal involved. The idea is to just "bullshit" about non-work stuff for a bit. :) [18:31:01] ah ha :-) [18:31:08] https://www.urbandictionary.com/define.php?term=shoot%20the%20shit [18:51:37] 10Jade, 10Scoring-platform-team (Current), 10User-Testing, 10Design: Jade Wireframes: Diff view controls - https://phabricator.wikimedia.org/T212377 (10Halfak) a:03Halfak [18:51:45] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity view mode - https://phabricator.wikimedia.org/T212379 (10Halfak) a:03Halfak [18:51:54] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity edit mode - https://phabricator.wikimedia.org/T212374 (10Halfak) a:03Halfak [18:52:52] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity edit mode - https://phabricator.wikimedia.org/T212374 (10Halfak) [19:28:33] accraze, how's tuning? [19:28:47] Does it look like we'll have a model to bring to beta soon or did something get jammed? [19:28:59] * halfak got lost in Jade docs for a while. [19:29:39] still tuning... been running for almost 1.5 hr, so maybe almost done? [19:30:37] Looks like we're still maxing out CPU usage. So we're probably going to miss our window. This is probably because I multiplied our grid by 4 with a recent change to editquality. That's not a big deal. [19:30:46] We'll aim to get this out to beta instead. [19:31:17] Now our only deadline is that I disappear in 2 hour :D [19:32:13] ok sounds good, how long does tuning a revert model generally take? [19:32:54] Used to be about an hour. I guess it might take 4 hours now :| [19:33:04] We might be able to trim the tuning grid. [19:33:12] And get some of that back :) [19:33:35] I'd just need to scan our winning tunings to look for configurations that never yield a good model. [19:40:51] whoa i think it just finished [19:40:55] \o/ [19:41:05] Wow! It did! [19:41:10] ok lemme update that PR real quick [19:41:15] Cool. :) [19:45:28] * halfak hammers F5 on the PR :) [19:45:59] Side note, I added next week to our https://etherpad.wikimedia.org/p/scoring_current_work notes and put a link to my summary of Jade product direction stuff there. [19:46:15] It's hacky but I hope it helps make Jade more straightforward. [19:47:42] cool, i'll take a look! [19:48:11] also PR is updated [19:48:13] After reviewing the state of Jade product documentation, I have renewed excitement for getting a coherent direction together. [19:48:16] Awesome. [19:48:33] Not bad! [19:48:39] This looks pretty good actually! [19:49:01] \o/ [19:49:03] * halfak sees the GaussianNB at the top of the list. [19:49:06] OK that's bad. [19:49:07] Hmm. [19:49:12] ahh... [19:50:01] I bet it is our ROC AUC statistic calculation. Something is screwy with it. But the fit of the Gradient Boosting model looks solid. [19:50:04] "learning_rate=0.01, max_depth=5, n_estimators=700, max_features="log2", min_samples_leaf=3" [19:50:07] Is a common best case. [19:50:30] Basically, "learn slowly, use lots of sub-models, and try not to overfit" [19:51:13] Hmm. It looks like the model did not get picked up by git lfs. That's frustrating. [19:51:17] "Binary file not shown." [19:52:02] I expected to see "Git LFS file not shown" [19:52:10] So! Time to dig. [19:52:26] https://github.com/wikimedia/editquality/blob/master/.gitattributes [19:52:34] This suggests that it *should* have been picked up. [19:54:22] accraze, I'm looking at "/home/accraze/editquality" and I don't see a .git directory. [19:54:56] That seems weird. [19:55:13] ahh yeah.... I scp'd everything this morning before I knew about the proxy stuff [19:55:25] Aha! [19:55:32] Maybe you don't have git LFS locally then? [19:55:43] possiby---checking [19:56:07] yeah that's it [19:56:23] i can do a `git rm` and re-push the model [19:56:50] Make sure to amend the commit so we don't have a commit with a big binary in it. [19:57:07] good call [19:57:52] Oh! looks like amend won't work [19:57:55] accraze, ^ [19:58:10] you need to instead "git reset HEAD~1" and then try the commit again. [19:58:19] You'll need to force-push too. [19:58:40] ahh ok makes sense [20:01:28] ok lfs worked and travis is running again [20:02:44] Once it finishes give me a call and I'll help walk you through the beta deploy process. :) [20:57:55] (03PS1) 10Accraze: adds glwiki revert model, info and tuning [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/520633 [20:59:20] (03CR) 10Halfak: [V: 03+2 C: 03+2] adds glwiki revert model, info and tuning [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/520633 (owner: 10Accraze) [21:28:41] (03PS3) 10DannyS712: Add help links to special pages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/520470 (https://phabricator.wikimedia.org/T226938) [21:31:52] (03CR) 10jerkins-bot: [V: 04-1] Add help links to special pages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/520470 (https://phabricator.wikimedia.org/T226938) (owner: 10DannyS712)