[00:04:51] I'm still not having any luck pulling the submodules for the production repo... [00:09:30] Success. I had a stale ssh key in Diffusion. [00:21:09] until tomorrow! [06:41:24] 10Scoring-platform-team-Backlog, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Grant AWight accounts on ores production clusters - https://phabricator.wikimedia.org/T168442#3364601 (10ArielGlenn) >>! In T168442#3380801, @RobH wrote: > Addition to the ores-admins is a sudo group, and thus will re... [08:48:38] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3381458 (10Sumit) Damaging: ``` make models/rowiki.damaging.gradient_boosting.model cat datasets/rowiki.lab... [09:40:59] 10Scoring-platform-team, 10Wikilabels: [Discuss] Wikilabels routes refactor - https://phabricator.wikimedia.org/T165046#3381600 (10Pginer-WMF) Thanks for the clarifications @Halfak Regarding the "workset" concept I wonder if that unit of work is defined because it is convenient for the machine (i.e., needed... [09:47:31] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3381611 (10Tgr) I have no idea what's going on there, the resource `pip_install_revscoring_dependencies_hack` should manually require... [13:42:44] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3382231 (10Sumit) Goodfaith: ``` make models/rowiki.goodfaith.gradient_boosting.model... [14:54:26] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3382520 (10Sumit) need to retrain the models after the regex update, PR soon. [15:23:02] o/ [15:23:28] FYI: out sick today. Food poisoning. Just here for a bit to do some bureaucratic stuff. [15:25:02] halfak: get well soon :) [15:27:52] Thanks dude [15:29:12] halfak: ugh; not fun. Rest well. [15:30:39] Thanks. [15:30:45] Will do my best :) [15:31:57] 10Scoring-platform-team, 10ORES, 10revscoring, 10artificial-intelligence: Why don't timeouts work during long regular expression matching? - https://phabricator.wikimedia.org/T168965#3382708 (10Halfak) [15:34:25] halfak: ouch, get well soon! [15:57:28] 10Scoring-platform-team-Backlog, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Grant AWight accounts on ores production clusters - https://phabricator.wikimedia.org/T168442#3382836 (10awight) :) @Halfak, I believe that's for you. [16:07:26] 10Scoring-platform-team-Backlog, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Grant AWight accounts on ores production clusters - https://phabricator.wikimedia.org/T168442#3382919 (10Halfak) Confirmed! [16:17:23] 10Scoring-platform-team, 10MediaWiki-JobQueue, 10ORES, 10Performance-Team, and 5 others: Job queue corruption after codfw switch over (Queue growth, duplicate runs) - https://phabricator.wikimedia.org/T163337#3382977 (10elukey) Experiment finished and nothing changed in rdb2003, so we can rule out that (re... [16:20:03] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3383011 (10awight) That sounds reasonable. I tried provisioning again just to get a clean bug report, using mediawiki-vagrant commit... [16:27:59] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3383129 (10Tgr) >>! In T159105#3383011, @awight wrote: > SSL succeeds this time, but we run into the problem I documented above: Did... [16:28:51] tgr: o/ my fault, I didn't realize there was an active patch. Looking through Gerrit now... [16:29:38] This one? https://gerrit.wikimedia.org/r/#/c/361639/1 [16:30:00] 10Scoring-platform-team, 10Wikilabels: [Discuss] Wikilabels routes refactor - https://phabricator.wikimedia.org/T165046#3383143 (10Halfak) The quick answer is **both**! There's a UX theoretical reason why people like doing work in chunks that take less than 5 minutes (see [some of my own research](https://arx... [16:30:19] awight: no, https://gerrit.wikimedia.org/r/#/c/344814/ [16:30:50] ooh--okay great, patching and starting from scratch... [16:32:38] tgr: btw, I found this amazing puppet option for debugging dependencies, https://docs.puppet.com/puppet/4.10/configuration.html#ordering [16:32:55] I set it to random and shake like a magic eight ball. [16:33:44] neat [16:34:26] It does feel a bit silly to fight randomness with randomness :) [16:34:43] I suspect we have lots of ordering bugs though, very few roles have been tested e.g. with enabling the role before the initial provisioning of MediaWiki [16:36:00] so it might mostly uncover programming errors which do not have real user impact (as long as users follow the standard workflow, anyway) [16:46:57] good point! I find that it's very common to assume that everything in a module executes in sequence. However, that seems to be an extremely risky practice, cos it falls apart as soon as any cross-module dependencies come into play. [17:02:59] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3383360 (10awight) It behaved better with the patch, first giving the SSL error but on the second provision, I got: ``` ==> default:... [17:07:17] 10Scoring-platform-team-Backlog, 10Design: Discuss and create a UI mockup for the Meta-ORES editor interface - https://phabricator.wikimedia.org/T168993#3383394 (10awight) [18:22:12] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3383898 (10awight) > Perhaps we're running the system python when trying to start the service? My guess was wrong, the stack trace i... [18:25:05] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:25:26] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:26:04] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:26:23] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:27:02] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:27:07] those are legit ^^ [18:27:12] they happen to me too [18:27:21] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:27:51] for some reason, "puppetmaster has issues" always makes me think of, https://en.wikipedia.org/wiki/Being_John_Malkovich#/media/File:Being_John_Malkovich_poster.jpg [18:28:01] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:28:10] paladox: Thanks for the heads-up! [18:28:18] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:28:21] Your welcome :) [18:28:59] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:29:16] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:29:32] ooh the notification interval is still 2 minutes... [18:29:47] we can up it [18:29:58] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:30:13] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:30:56] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:31:11] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:31:21] check_interval = 7m [18:31:21] retry_interval = 4m [18:31:29] how long do we set it for ? [18:31:31] awight ^^ [18:31:55] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:32:08] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:32:53] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:33:06] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:33:29] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:33:50] awight https://gerrit.wikimedia.org/r/#/c/361714/2 [18:33:52] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:33:56] though i doint think it will work [18:34:01] puppet is broken [18:34:03] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:34:23] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:34:26] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:34:50] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:35:00] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:35:21] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:35:24] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:36:24] IMO notification interval should be something like 60m, or 0 (never). paladox: What do you think? [18:36:36] 60m? [18:36:46] that's 60 minutes [18:37:02] yup [18:37:11] hmm i could do that [18:37:15] i guess that's retry [18:37:22] check_interval = 4m [18:37:22] retry_interval = 50m [18:37:23] ? [18:38:38] I think [18:38:39] https://docs.icinga.com/icinga2/latest/doc/module/icinga2/toc#!/icinga2/latest/doc/module/icinga2/chapter/monitoring-basics#alert-notifications [18:38:52] template Notification..... { interval = .... [18:38:58] oh i see [18:39:01] retry_interval = 50m [18:39:10] That's probably something different [18:39:17] that retrys [18:39:20] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3383960 (10Sumit) https://github.com/wiki-ai/editquality/pull/78 [18:39:21] so for example [18:39:33] it will send a notification every 50 mins [18:39:34] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:39:39] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:39:39] since it will retry every 50 mins [18:39:41] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:39:44] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:40:34] paladox: retry_interval seems to be how often icinga checks the service, so that should probably be nice and short like 1m [18:40:44] ah i see [18:40:49] lol i set it higher [18:40:50] But is that a config option in icinga2? [18:40:52] i will change that [18:40:54] yeh [18:41:02] Can you remind me where your config lives? [18:41:23] https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/monitoring-basics [18:41:33] * paladox gets url [18:41:53] awight: https://gerrit.wikimedia.org/r/#/admin/projects/labs/icinga2 [18:41:56] ty! [18:42:07] https://github.com/wikimedia/labs-icinga2 [18:42:11] your welcome :) [18:43:23] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:43:33] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:43:36] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:43:42] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:46:34] paladox: Feel free to add me to https://gerrit.wikimedia.org/r/#/admin/groups/1332,members when you get the chance... [18:46:48] awight i've added you i think [18:46:54] since i can add only groups [18:46:57] i've added ores [18:47:06] cool, I see now [18:47:12] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:47:22] https://gerrit.wikimedia.org/r/#/admin/groups/1063,members [18:47:26] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:47:31] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:47:40] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:50:51] srrodlund: Hi! How's this for dry copy? https://www.mediawiki.org/wiki/Meta-ORES [18:50:57] I need to human that up [18:51:01] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:51:07] any suggestions are welcome! [18:51:20] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:51:25] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:51:38] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:54:50] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [18:55:13] RECOVERY - puppet on ores-web-05 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:55:20] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:55:36] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:59:15] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:59:35] PROBLEM - puppet on Ores-Compute-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:03:10] RECOVERY - puppet on ores-worker-05 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:03:11] will non english speakers read this? [19:03:33] RECOVERY - puppet on Ores-Compute-01 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:03:57] aka -- who is your target audience? [19:04:27] I think, everyone from zero and up :) [19:04:41] Yeah non-English, and non-technical as well [19:05:32] srrodlund: ^ [19:05:48] I'd like this to be the entry point for anyone interested in the problem [19:06:13] ... which is really interesting. Just don't tell the machines that we're trying to help humans do battle with AIs [19:11:58] codezee: Hi! Let me know if it's still helpful that I try to run the draftquality model pre- and post- sentiment patch? [19:12:22] I'd recommed that it is simplified then [19:13:15] codezee: I found some surprising things about that makefile yesterday, mostly that it's training on a 800k set rather than a sample like the others. But halfak thinks that's actually the right thing to do... [19:14:11] srrodlund: Okay, thanks I'll do that. Yeah I don't even quite know what "robust false-positive" means, I was just copy and pasting :) [19:14:54] okay why don't we talk about this? I can give you some pro-tips. Not that I know anything ;-) [19:15:26] srrodlund: You think it's fair to simplify but expand? I feel like a long story about the context would be really helpful for motivating the project. [19:15:37] lol you know thing [19:23:29] awight: "a sample like the others" what others you mean? [19:25:32] I think editquality is mostly using 20k sample sizes [19:28:57] awight: editquality has only that many samples in general i think bec of labeling constraints but many more labeled drafts are present i think if more drafts are present its ok to test on those many, we always have the benefit of cross validation [19:29:23] do you have an alternate viewpoint on why we should have less samples? [19:29:57] awight: also, wrt to above query, its ok if you report the score of draftquality post-sentiment patch [19:30:09] i think pre-sentiment patch report is already there? [19:30:49] I'm chronically undereducated about machine learning, so my only concern was that 800k samples takes much longer and more memory. But it makes sense if the result is going to be better... [19:31:33] Great, thanks for saving me the pre-patch run. I'll go ahead and train the post-patch model. [19:34:25] awight: but you know trying to know the learning curve as a function of the data size would indeed be an interesting exercise i think... [19:34:50] see https://stats.stackexchange.com/questions/51490/how-large-a-training-set-is-needed [19:36:22] also i was wondering why we don't have a gerrit like bot that pastes PRs made on github on phabricator... [19:36:38] codezee: Do you happen to have a link to the pre-sentiment patch? [19:36:58] awight: the pre-sentiment patch is master isn't it? [19:37:09] codezee: Good thought--probably because there's a slowly simmering religious war about choice of tooling... [19:37:27] codezee: yes, I just don't know off-hand where we keep the latest test stats [19:38:00] awight: if you want results, here - https://github.com/wiki-ai/draftquality/tree/master/tuning_reports [19:38:27] max is around 84-85% i think [19:39:23] oho! awesome. [19:42:01] Argh, I'm having trouble installing py dependencies for some reason. I'll need lunch to make progress on this :-) [22:02:16] Nettrom: I am curious, did you use ORES utilities for doing the experiment on https://pdfs.semanticscholar.org/8cf3/9ea9233bf72577d54af0845711288c4118fd.pdf ? [22:03:36] glorian_wd: the wikiclass library, which drives ORES wp10 model, is built upon the model from that paper (and from our CSCW 2015 paper, see the Appendix of http://www-users.cs.umn.edu/~morten/publications/cscw2015-improvementprojects.pdf) [22:03:54] the 2013 paper was done in WEKA [22:06:30] Nettrom: gotcha. Thanks :) [22:50:39] 10Scoring-platform-team, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels, 10Patch-For-Review: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3384630 (10Tgr) pytz 2012c was required by revscoring [[https://github.com/wiki-ai/revscoring/commit/68d3c5456162638dfb5e528a4d1f6a64... [22:57:16] Oh. Now I'm finally up against that permissions error [22:57:17] mwapi.errors.APIError: permissiondenied: You don't have permission to view a page's deleted history. -- None [22:57:42] * awight gnashes teeth