[07:40:31] is it normal to see bot edits in goodfaith/damaging edit labeling? [08:00:03] also, how often are translations updates? for hu, the yes/no -> damaging/not damaging thing was fixed two weeks ago but still not visible [12:09:47] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, 10Scap: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3827656 (10akosiaris) [13:53:51] (03PS1) 10Ladsgroup: Introduce ScoreParser class [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397548 (https://phabricator.wikimedia.org/T181334) [14:06:43] Halfak mornig [14:06:47] Morning* [15:03:06] o/ akosiaris [15:03:14] Did you have a chance to see my response in https://gerrit.wikimedia.org/r/#/c/396064/ ? [15:03:26] * akosiaris in a meeting currently, will do later [15:04:31] kk thanks [15:12:24] PROBLEM - puppet on ores-worker-06 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:14:59] PROBLEM - puppet on ores-worker-08 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:43:14] RECOVERY - puppet on ores-worker-06 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:44:34] RECOVERY - puppet on ores-worker-08 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:44:44] 10Scoring-platform-team (Current), 10ORES: id string in ORES does not include "features" - https://phabricator.wikimedia.org/T179509#3828297 (10Halfak) a:03awight [15:51:08] 10Scoring-platform-team (Current), 10Wikimedia-Incident: How can we test all the wiki/page combinations that can be affected by ORES? - https://phabricator.wikimedia.org/T181830#3828305 (10awight) a:05awight>03None [15:51:27] 10Scoring-platform-team (Current), 10Research Ideas, 10drafttopic-modeling, 10artificial-intelligence: Revscoring: Statistic for multilabel classification - https://phabricator.wikimedia.org/T181166#3828307 (10Sumit) https://github.com/wiki-ai/revscoring/pull/376 [15:52:18] 10Scoring-platform-team, 10Research Ideas, 10drafttopic-modeling, 10artificial-intelligence: Revscoring tune does not recognize a set of labels as target - https://phabricator.wikimedia.org/T181163#3828314 (10Sumit) https://github.com/wiki-ai/revscoring/pull/376 [15:53:31] 10Scoring-platform-team (Current), 10Wikimedia-Incident: How can we test all the wiki/page combinations that can be affected by ORES? - https://phabricator.wikimedia.org/T181830#3828316 (10awight) [15:57:18] 10Scoring-platform-team (Current), 10Wikimedia-Incident: How can we test all the wiki/page combinations that can be affected by ORES? - https://phabricator.wikimedia.org/T181830#3828322 (10Halfak) * We could do browser testing on a schedule or on a commit (in ORES service or Ext.) * We could have a QA speciali... [16:05:05] 10Scoring-platform-team, 10Global-Collaboration, 10MediaWiki-extensions-ORES: Clarify that ORES UI can be disabled separately from the extension - https://phabricator.wikimedia.org/T182600#3828356 (10awight) [16:13:55] halfak: mind if I make akosiaris’s suggested changes to the uWSGI config? [16:18:28] awight, if akosiaris agrees to help clean up any mess that results. [16:18:29] :) [16:18:40] lol I think that’s his job [16:18:58] * awight dodges cybertomatoes [16:19:15] * apergos tomatoes the spots on awight's left AND right [16:19:17] :-P [16:19:38] a thorough salad! [16:19:53] we do our best [16:24:00] awight: ping for the ext. review :D [16:24:17] Amir1: :) I’m on it [16:24:33] I've got to go shovel. [16:24:35] back in a bit. [16:30:49] wiki-ai/revscoring#1350 (master - 63bc9fb : Amir Sarabadani): The build has errored. https://travis-ci.org/wiki-ai/revscoring/builds/314866101 [16:40:24] I need to fix revscoring [16:46:34] wiki-ai/revscoring#1351 (travis - 455f57a : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/314872947 [16:46:47] awight: ^ this fixes travis [16:50:18] halfak: a bit quick on the merge button! [16:50:37] Oh? did I miss a problem? [16:51:15] halfak: well, I had just approved it so was thinking I would be the one pressing the next button :) [16:51:19] There is a problem though [16:51:53] It doesn’t look like travis is even running that line—and also, if we were using wheels we probably wouldn’t need this workaround in the first place [16:52:36] Amir1: Do you have any idea how travis is skipping machine setup? Are we reusing VMs? [16:52:54] wiki-ai/revscoring#1353 (master - fa9a62d : Aaron Halfaker): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/314875993 [16:53:18] awight, I thought the same about wheels. We should 'pip install pip --upgrade' and maybe it'll then pull down the wheels. [16:53:29] yeah that’s probably the issue. [16:54:17] It definitely ran the line: https://travis-ci.org/wiki-ai/revscoring/builds/314872947 [16:54:19] line 813 [16:54:27] O/ [16:54:56] halfak: pip is already 9.0.1 [16:55:14] ahh. Maybe there just aren't wheels for the old versions we are using? [16:58:06] halfak: I’m following that same breadcrumb :) [16:58:36] kk [16:58:43] * halfak continues to review codezee's work [16:58:45] They bumped the major version! [16:58:58] hehe sorry I wasn’t trying to chase you away that time. [17:00:12] The wheels are still available from pypi, https://pypi.python.org/pypi/scipy/0.17.1 [17:00:19] awight, we might be able to do an upgrade. [17:00:23] :P still minor version bump. [17:00:28] not 1.0 yet [17:00:57] Im sorry whats the issue? [17:02:45] codezee, I just finished reviewing https://github.com/wiki-ai/revscoring/pull/376 [17:03:00] I can do a follow-up commit if you're busy on something else.. [17:03:41] halfak: i'm looking [17:04:10] halfak: regarding --multilabel, what do you mean by "adding in parameter config?" [17:04:33] codezee, the parameters that tune searches. [17:04:52] [17:05:00] The first argument to the utility [17:05:20] i see [17:08:38] Amir1: I’m confused about why travis_wait or any of this direct pip package installation is needed. [17:08:53] The times are 5s for numpy, 10s for scipy, and 40s for the rest of the modules [17:09:22] Travis docs suggest that the default timeout is 10 minutes: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received [17:09:31] awight, used to compile scipy [17:09:37] scipy takes a long time to compile. [17:09:38] and that’s with no stdout. pip install stdouts like crazy [17:09:41] then we'll have to yield an additional parameter in _model_param_grid and pass it to _cross_validate [17:09:44] halfak: lemme check that. [17:09:53] codezee, I think so. [17:10:30] Alternatively, we could allow for the -p "param=value" pattern that we use in the training utilities. [17:10:35] Amir1: Do you happen to remember a CI build where we timed out? [17:10:37] so, -p "multlabel=true" [17:10:40] I’d like to see what happened. [17:10:42] *i [17:11:02] awight, it's been a long time, I think. Probably 2 years. [17:11:03] \me looks [17:11:10] um why are we making this fix, then? [17:11:32] Oh good Q. [17:11:33] It just crufts up the .travis config with unnecessary stuff? [17:11:35] awight: I remember, it was the times that installing numpy and scipy took so long that it times out [17:11:37] * halfak waits for Amir1 to look [17:12:10] so we added these options [17:12:12] Amir1: The only reason I’d push back on this is, now we have to update the .travis.yml every time we increment the library versions, or else we’re silently testing against the wrong versions. [17:13:21] awight: I'd love to see a PR cleanup everything therre [17:13:36] Challenge accepted ;-) [17:13:42] :P [17:14:43] halfak: so adding an option to do -p "multilabel=True" and setting that in additional params, sounds fair? [17:14:59] +1 [17:16:03] actually.... not that I look at it, it seems to me that having a 'multilabel' param works. [17:16:18] Sorry for the confusion. I'm looking at "center" and "scale" etc. [17:16:33] I think just dropping the label-auto-detection makes sense. [17:21:26] codezee, ^ [17:21:31] sorry for the confusion [17:21:49] I should have looked at the whole file before commenting [17:22:21] I'm back to advocating for `--multilabel` but dropping get_multilabel_set. [17:23:49] halfak: if we're dropping auto-detection of labels then we're passing them in cli, but that seems counter-intuitive since there are some 40 labels [17:23:56] we'll end up writing a long command [17:24:37] plus additional logic outside tune to figure that out [17:24:56] wiki-ai/revscoring#1356 (cleanup_travis - 266fbe9 : Adam Roses Wight): The build has errored. https://travis-ci.org/wiki-ai/revscoring/builds/314892572 [17:25:07] codezee, agreed on the verbosity issue. [17:25:20] codezee, the logic already exists if you look at that line I linked to. [17:25:35] We'll need to specify population rates eventually anyway. [17:26:16] You could define a file type (e.g. a yaml file where each key is a potential label and values underneath the key describe pop rate, weight, etc.) [17:26:50] e.g. `stem.mathematics: {'population_rate': 0.03, 'weight': 0.10}` [17:27:35] That could replace --labels, --label-weight, and --pop-rate [17:28:23] See line 117 in your config. [17:28:34] *docopt parsing [17:28:44] yeah i got the line... [17:30:59] (03CR) 10Awight: [C: 04-1] "This is a great cleanup. My only question is whether HashModelLookup can be moved to the tests directory." (036 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/395811 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:31:20] halfak: so you're suggesting to add this mapping to or we should have a new parameter? [17:31:35] that reads from a yaml file [17:33:10] wiki-ai/revscoring#1357 (cleanup_travis - a77b39f : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/314897290 [17:33:15] codezee, I think it should be like "--labels-config" since most of the time, we'll just want to use "--labels" or "--pop-rates" [17:33:22] But yeah, that option. [17:35:18] Amir1: Thanks for these incremental patches, it’s a pleasure to review! [17:36:16] Cool [17:37:15] halfak: i see, but don't you think that the automatic labels deduction is a nice fallback feature when labels-config is not specified. Agreed that we'll not use it but for a newbie who just wants to play with the data, they'll not have to go through the burden of specifying pop-rates for each 40 labels or write an external script to first list labels in that format? [17:37:32] we can run it conditionally if labels are not supplied [17:37:58] like even now testing was a breeze for me bec i didn't have to write a script to list labels first, it did on its own [17:38:10] Thanks for the review [17:45:06] (03CR) 10Ladsgroup: Join decomposition of ores_model table queries (035 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/395811 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:48:11] (03CR) 10Awight: [C: 032] "I sort-of buy the argument that HashModelLookup might be used in the wild :)" (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/395811 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:49:10] (03CR) 10Awight: [C: 032] Introduce ScoreParser class (034 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397548 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:49:37] And tests! Awesome. [17:51:15] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, 10Performance: Diagnose and fix 4.5k req/min ceiling for ores* requests - https://phabricator.wikimedia.org/T182249#3828705 (10awight) [17:51:21] (03Merged) 10jenkins-bot: Join decomposition of ores_model table queries [extensions/ORES] - 10https://gerrit.wikimedia.org/r/395811 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:51:29] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, 10Performance: Diagnose and fix 4.5k req/min ceiling for ores* requests - https://phabricator.wikimedia.org/T182249#3817970 (10awight) p:05Triage>03Normal [17:51:34] Thanks for merging [17:51:36] \o/ [17:52:05] of course! [17:52:32] codezee, was deep in emails. Sorry to miss your ping. [17:52:34] * halfak reads [17:52:38] Amir1: Sorry it looked like I had disappeared. Also, I don’t work Fri-Sun :p [17:52:56] (03Merged) 10jenkins-bot: Introduce ScoreParser class [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397548 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:53:17] codezee, I've been bitten a lot by not having the labels we expected represented in the data. [17:53:25] there's a reason that those checks exist. [17:53:41] 99% of the time, this is not burdensome. [17:54:00] oh, so you're saying that supplying labels is more of a guarantee... [17:54:26] I forgot, thanks for reminding me [17:56:32] codezee, right. I think that, in this case, it's very good to be explicity. [17:56:40] *explicit [17:59:37] halfak: last thing, it should be a required param or optional like --labels ? [18:00:06] codezee, the system will error out if --labels --pop-rates or --label-weights aren't provided. [18:00:27] So I think it should fit into that OR-based requirement. [18:01:12] oh,ok [18:01:29] * looking for that logic [18:03:26] It's in utilities/util.py [18:04:34] (03PS1) 10Ladsgroup: Clean up the model lookup code for ApiHooksHandler a bit [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397586 [18:04:49] awight: Implemented your suggestions [18:04:50] Thanks [18:05:26] (03CR) 10Awight: [C: 032] Clean up the model lookup code for ApiHooksHandler a bit [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397586 (owner: 10Ladsgroup) [18:05:56] i was on the same func for so long and didn't see the RuntimeError at last :P [18:07:14] (03Merged) 10jenkins-bot: Clean up the model lookup code for ApiHooksHandler a bit [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397586 (owner: 10Ladsgroup) [18:09:14] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Train/test reverted model for Catalan Wikipedia - https://phabricator.wikimedia.org/T182611#3828788 (10Halfak) [18:09:43] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Implement language support for Catalan - https://phabricator.wikimedia.org/T182612#3828800 (10Halfak) [18:10:08] * halfak --> lunch [18:24:23] 10Scoring-platform-team, 10ORES, 10Operations: Investigate why ORES logs are being written to syslog despite explicit logging config - https://phabricator.wikimedia.org/T182614#3828863 (10awight) p:05Triage>03Normal [18:26:12] 10Scoring-platform-team, 10ORES, 10Operations: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614#3828863 (10awight) [18:37:56] how does translation update in wikilabels work? [18:38:11] does it depend on manual deploys? [18:48:15] (03PS1) 10Ladsgroup: Use ScoreParser instead of Cache::processRevision in ApiHooksHandler [extensions/ORES] - 10https://gerrit.wikimedia.org/r/397597 (https://phabricator.wikimedia.org/T181334) [19:04:45] halfak: sorry for the confusion, i got stumped bec of a bug in read_labels_and_population_rates so even if I was supplying none of labels or pop rates I could get revscoring to continue [19:04:48] the bug is at - https://github.com/wiki-ai/revscoring/blob/master/revscoring/utilities/util.py#L53 [19:05:20] label_weights_strs or pop_rates_strs is a "[]" so its never None [19:06:18] i'll fix it in this change [19:10:34] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, 10Scap: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829035 (10mmodell) @awight: can you try it now with the -v flag, scap should include the ssh logs in verbose output now. [19:12:04] Nice thanks codezee :) [19:17:02] i think its good to go now :) [19:20:52] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, 10Scap: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829075 (10awight) Done. Logs are in scap-sync-2017-12-09-0004.log (sic., note that the dates are still misleading) Thanks for t... [19:28:34] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, 10Scap: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3829109 (10mmodell) Something really strange is going on... ``` debug1: Offering RSA public key: /etc/keyholder.d/deploy_servic... [19:32:15] runtime even with word2vec looks huge, tune has been running since 3 hrs and still running on full dataset.... [19:35:21] halfak: im seeing on icinga logs that one of the ores hosts on cloud vps had an error, did you guys see that/fix it or what was the deal? [19:35:53] Zppix: A bunch of cloud stuff had the same error, so I assume it’s just the end of the world, right on schedule. [19:36:06] awight: lmao ok just checking up [19:36:41] Catch you on the flip side! [19:38:33] 🤦🏻‍♂️ [19:46:39] codezee, strange. How quickly can you train one model? [19:51:20] halfak: i'm not sure, let me check with cv_train [19:51:37] hopefully i shouldn't need to change anything in cv_train [19:53:01] halfak: should I use cv_train or just write a small script that trains? [20:10:12] nvm started a cv train with 3 folds [20:33:49] leaving .... o/ [21:27:21] \o/ [21:27:27] sorry was in meeting :| [21:28:52] Why do yall have so many meetings [21:33:00] Got a lot of projects. I met a lot today to talk about improving documentation. [21:33:15] And a little bit to organize what's going on with the people working on ORES stuff. [21:34:40] If you write tasks on docs (smaller tasks) i can mentor them for GCI [21:36:43] Zppix, hard to do that one since it tends to require deep knowledge to write docs. [21:38:18] Hmm... [21:38:31] Ill try to think of somethig [22:02:45] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10Regression, and 4 others: OresDamagingPref back-compatibility is logging exceptions - https://phabricator.wikimedia.org/T182354#3829539 (10greg) [22:03:41] Amir1: ^ Can we close this? [22:04:23] awight, did you ever get that puppet patch worked out with ako*? [22:04:58] halfak: It’s rewritten to his recommendations, and I pinged him. Didn’t hear back yet. [22:05:04] kk [22:05:17] btw you should subscribe to https://phabricator.wikimedia.org/T182614 [22:05:22] I'll go have a look to and see if I can +1 [22:05:46] cool [22:08:36] {{done}} [22:08:36] How cool, halfak! [22:25:28] OK I updated the JADE PR too [22:25:36] With that I think I'm going to call it a day. [22:25:55] :) I started it on the bike and I'm ending it on the bike. Getting better and better about pedaling and working.