[08:09:46] 10Scoring-platform-team, 10Wikilabels: [Discuss] Wikilabels routes refactor - https://phabricator.wikimedia.org/T165046#3385250 (10Jan_Dittrich) > chunks that take less than 5 minutes And aside of the timeframe: It probably gives a sense of achievement to have finished a workset (or however we shall call it i... [12:36:16] 10Scoring-platform-team-Backlog, 10Labs, 10Labs-Infrastructure: Keep wmflabs scoring boxes up-to-date - https://phabricator.wikimedia.org/T168478#3386243 (10faidon) [14:01:23] o/ [14:01:28] * halfak works through his email [14:01:34] o/ [14:01:38] Still kinda sick today but feeling a lot better [14:02:25] halfak: does it help to drink more water? perhaps, it could replenish your body [14:02:48] Yeah. I'm pretty good with anatomy and physiology literacy. I [14:03:04] 've been doing juice because it helps keep water and electrolytes/blood sugar [14:59:27] halfak: is it possible for you to get to https://github.com/wiki-ai/wikiclass/pull/39/files today? Thanks :) [15:47:57] Amir1, https://phabricator.wikimedia.org/T108440#3385399 [15:47:58] :/ [15:48:04] glorian_wd, likely yes [15:48:21] halfak: alright thanks :D [16:30:44] Hey folks. Anyone got a lot of experience with AbuseFilter? [16:43:43] not a lot but some [16:46:51] I've been talking to Caroline Sinders and I'm not very familiar with it at all. Was hoping to find someone who was willing to talk to her or who might be able to make a better introduction :) [16:47:06] Caroline Sinders == Researcher working on anti-harassment stuff. [16:47:42] this paper might be interesting with regards to those of us interested in understanding what DNNs do: https://arxiv.org/abs/1706.08606 (disclaimer: I haven’t read the paper yet) (see also: https://deepmind.com/blog/cognitive-psychology/?utm_content=56704112) [17:03:26] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train/test damaging & goodfaith models for Albanian Wikipedia - https://phabricator.wikimedia.org/T163009#3387654 (10Halfak) [17:03:28] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3387655 (10Halfak) [17:03:30] 10Scoring-platform-team: Scoring platform team FY18 Q1 - https://phabricator.wikimedia.org/T166045#3387653 (10Halfak) [17:06:52] halfak: it seems MusikAnimal made about 30% of all abusefilter changes in 2017 [17:07:34] (on enwiki) [17:08:36] Oh cool. Thanks for looking that up. :) [17:09:34] https://phabricator.wikimedia.org/P5645 [17:27:04] nice. Thanks :)_ [17:34:52] halfak: :/ [17:35:05] * halfak got kinda ranty [17:35:31] Where does this "I don't want to depend on any libraries" culture come from [17:35:43] Other people's code is great! I love not having to maintain everything [17:43:07] Amir1, do you have any use for mediawiki-ores? [17:43:27] The VM [17:43:29] halfak: what is mediawiki-ores? [17:43:31] crikey! My draftquality training job is still running, currently up to something like 30 hours of CPU time. [17:43:35] oh [17:43:41] I think so let me check [17:43:49] awight, ha yeah. Bumping up the observations takes a lot of time. [17:44:00] We should cut the number of cross-validations. [17:44:08] halfak: we had it for public testing when there wasn't any beta cluster/ etc. [17:44:08] No reason to train 10 models [17:44:15] codezee and I were chatting, about how it would be nice to plot the learning curve for that [17:44:23] I have never used it in a very long tim [17:44:25] *time [17:44:32] kill it [17:44:43] Amir1, I'd like to kill ores-experiment too. I am trying to figure out how to make a bigger ores-compute-* [17:45:07] I use ores-experiment more often [17:45:12] when to test a PR [17:45:28] awight, one issue with cutting observations is that the uncommon classes become so uncommon as to not be learned. [17:45:35] Then we get into stratified sampling. [17:45:36] do you want to ask for more resource in labs? [17:45:51] Amir1, yeah. maybe. I'll leave ores-experiment if you use it. [17:45:53] They can increase the project capacity [17:45:54] We should get it humbered. [17:45:57] *numbered [17:46:05] I'll be dropping hashing-vector [17:46:14] Sabya hasn't been working on that in a LOOONG time. [17:46:31] IMO his notes are the primary value from that work. [17:56:02] Can I just turn cross-validation off for producing the draftquality numbers codezee needs? [17:56:07] * halfak presses "delete instances" [17:56:15] Yup. [17:56:22] k [17:56:27] I think the argument to "cv_train" is "--folds 1" [17:56:40] * awight ^C's 30 hours of cpu time [17:56:40] It might complain that you need at least 2 folds. [17:56:59] In that case you can run "train" instead of "cv_train" [17:57:15] With "train" it pulls in all the data to train, so you'll have to manually withhold a test set [17:57:38] needs folds, eh? I will distract the computer with all the folds it desires: http://www.giladorigami.com/P_MousePC_Guspath.jpg [17:57:38] *train_model [17:58:48] I'm going to try to set us up a super-big instance on ores-staging so that we can more easily train models with lots of data. [18:00:05] Amir1, could I replace ores-experiment with a single bigram instance that you could run tests on along with our model building? [18:00:08] Another instincts question--I was training the vanilla draftquality model to have a baseline to compare with codezee's patch, but can I just trust the existing stats here? https://github.com/wiki-ai/draftquality/blob/master/tuning_reports/enwiki.draft_quality.md [18:00:22] halfak: sure [18:00:52] awight, probably not. If you can download the repo/model to your machine, run "revscoring model_info " [18:01:07] awight, I wouldn't trust them but I wouldn't expect there to be much of a difference. [18:01:18] Also I don't think the tuning score is the best indicator :/ [18:01:28] hmm okay, I'll go ahead with the baseline run then [18:01:36] * awight quietly turns workers up to 4 [18:01:38] Amir1, cool. Anything you need from ores-experiment or is it OK to just delete when I'm ready? [18:01:53] nah, delete everything [18:01:56] cool [18:02:10] Any opinions on "ores-compute-*" as a name? [18:02:24] Like, maybe we call it "ores-build-*" or something like that [18:02:31] Maybe "ores-misc-*" [18:02:42] ores-misc looks fun [18:02:57] Last questions, I think--since his patch affects how features are built, I should recreate with_cache, correct? [18:03:00] ok. new name will be ores-misc-01 [18:03:11] awight, yes +1 [18:13:05] 10Scoring-platform-team-Backlog, 10Privacy: Use filesystem group permissions to protect deleted article content on ores wmflabs boxes - https://phabricator.wikimedia.org/T169123#3387876 (10awight) [18:15:35] Ah, if we're no longer limiting draftquality to 500k samples, then why would we shuffle? [18:15:57] Is anything about the training sequence-dependent? [18:22:10] awww it seems that training is single-threaded, only cross-validation takes advantage of multiple cores? [18:23:44] awight: https://gerrit.wikimedia.org/r/#/c/361920/ [18:24:16] paladox: rad! reading now [18:24:22] Thanks :) [18:26:54] that repo doesn't have any CI jobs, so I need to force submit, true? [18:27:23] thanks :) [18:27:24] yep [18:27:35] Already merged it after you did c+2 [18:27:58] Will deploy it now (it auto deploys but will take a while) might as well do it now :) [18:28:48] I'll break something major so we can test it ;-) [18:29:20] lol [18:29:22] haha [18:29:29] reboot something [18:29:33] it may hit the reboot [18:29:41] deployed :) [18:29:51] Don't worry, it's just a "fix" [18:30:06] :) [18:35:26] * paladox wonders what other changes to do :) [18:40:43] halfak: take yr time. but ping: 11:15 <+awight> Ah, if we're no longer limiting draftquality to 500k samples, then why would we shuffle? [18:46:45] awight: https://gerrit.wikimedia.org/r/#/c/361925/ :) [18:51:37] PROBLEM - check disk on ores-redis-02 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:46] PROBLEM - check disk on ores-web-04 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:47] PROBLEM - check disk on ores-worker-06 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:47] PROBLEM - check disk on ores-web-03 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:47] PROBLEM - check disk on ores-worker-09 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:47] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:51:49] PROBLEM - check disk on ores-worker-07 is CRITICAL: NRPE: Command 'check_disk' not defined PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 8 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:51:49] uh [18:51:50] PROBLEM - check disk on ores-worker-10 is CRITICAL: NRPE: Command 'check_disk' not defined [18:51:51] PROBLEM - check disk on ores-worker-08 is CRITICAL: NRPE: Command 'check_disk' not defined [18:52:01] lol [18:52:11] woops [18:52:16] lessee about the notification interval! [18:52:18] that check_disk is not meant to be there [18:52:25] haha [18:52:26] :) [18:52:45] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:52:46] i will test the vars changes you mentioned [18:52:47] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 9 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:52:54] we found puppet errors anyways ^^ [18:53:33] 10Scoring-platform-team: Minor cleanup in Makefiles - https://phabricator.wikimedia.org/T168904#3388046 (10awight) We can configure makefiles to delete the target on error by using the pseudo-target `.DELETE_ON_ERROR:`, see https://www.gnu.org/software/make/manual/html_node/Errors.html#Errors [18:53:43] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:53:47] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 10 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:54:46] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 11 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:54:46] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:55:45] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 12 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:55:45] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:56:43] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 13 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:56:44] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:56:54] Darn, the notification interval is still 2m [18:57:07] hmm [18:57:19] i think it's just puppet heh. [18:57:28] the check_disk seems to be holding [18:57:42] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 14 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:57:43] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:58:41] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 15 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:58:43] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [18:59:00] I am fixing check_disk to prevent those from erroring out [18:59:40] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 16 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [18:59:42] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:00:39] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 17 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:00:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:01:38] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 18 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:01:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:02:37] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 19 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:02:40] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 12 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:03:00] well those puppet notifications reallly know how to talk (lol) [19:03:36] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 20 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:03:39] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:04:52] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 21 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:04:53] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 15 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:05:51] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 22 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:05:52] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 16 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:06:48] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 23 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:06:51] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 17 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:07:47] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 24 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:07:49] woah [19:07:50] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 18 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:08:44] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 25 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:08:50] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 19 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:09:01] paladox, what's going on? [19:09:09] halfak we added new hosts [19:09:14] paladox: So I just noticed, all those notification intervals we changed are for email notifications. [19:09:18] and some hosts have failed puppet [19:09:20] oh [19:09:23] lol woops [19:09:35] check_disk should be fixed now. [19:09:43] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 26 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:09:50] hmm still shows the error [19:09:50] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 20 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:09:51] paladox: maybe try a "template" block like I was mentioning in the gerrit comment? [19:10:00] ACKNOWLEDGEMENT - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 26 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] paladox Ack [19:10:08] awight yep [19:10:13] will do that now [19:10:23] ACKNOWLEDGEMENT - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 20 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] paladox Ack [19:10:37] oh [19:10:38] i see [19:10:39] I see you're busy ;-) [19:10:41] icinga2 failed [19:10:47] haha :) [19:10:56] I haven't found the IRC relay settings yet... [19:12:06] found it [19:12:11] paladox: ooh. Those "mail" notifications are actually the irc bridge [19:12:18] yeh [19:12:21] I was wrong about them having anything to do w/ email [19:14:06] i need to rename them so they are less confusing :) [19:14:19] it's now done through a shell script instead of config file like icinga 1.x [19:15:25] damn, the backend is failing Backend icing is not running [19:15:44] :) We don't mind, at least while it is spamblasting [19:15:53] lol hahaahaha [19:16:13] oh no! your barking dog seems to have something stuck in its throat :p [19:16:34] lol yeh [19:17:06] oh [19:17:07] it fails [19:17:38] it's complaning about undefined [19:17:40] variables [19:18:43] ah [19:19:38] hehe sorry to randomize your config file there [19:19:56] works [19:20:03] randomize? [19:20:08] What do you mean? :) [19:20:33] That I had no need to quibble about the vars.*, if that might be related. [19:21:20] hmm, nope. I think i can remove the var. prefix as long as it is in host.config (though not sure about that yet) [19:21:47] someone should check the puppet errors on ores-web-04 and ores-worker-07 :) [19:22:07] K I'll take a look [19:22:23] thanks :) :) [19:25:35] it wants to downgrade ldap-utils, from 2.4.44+dfsg-5~bpo8+1 to 2.4.41+dfsg-1+wmf1 [19:25:46] I think this is caused by our random apt repo [19:25:56] ah yeh [19:26:03] i think that happened before :) [19:26:20] yeah I think the right solution is to get rid of our custom repo [19:28:13] yeh [19:28:41] awight did you press y for it to downgrade it ? [19:28:42] :) [19:28:51] not yet [19:28:55] I wanna kill that whole thing [19:29:00] ok [19:29:07] 10Scoring-platform-team-Backlog: Remove custom apt repo from ores labs boxes - https://phabricator.wikimedia.org/T169129#3388212 (10awight) [19:29:34] halfak: Amir1: ^ I think we chatted about this already, and the custom repo is just cruft? [19:30:15] Wat [19:30:23] Custom repo? [19:30:42] Oh an apt repo? [19:30:56] Yeah, I think yuvi was testing a custom apt repo at one point. [19:31:12] If that's what you're looking at (I wouldn't know for sure either) then it can definitely go. [19:31:23] * halfak pastes in phab [19:31:52] 10Scoring-platform-team-Backlog: Remove custom apt repo from ores labs boxes - https://phabricator.wikimedia.org/T169129#3388230 (10Halfak) Yeah, I think @yuvipanda was testing a custom apt repo at one point. If that's what you're looking at (I wouldn't know for sure either) then it can definitely be removed. [19:32:10] OMG I ate a slice of pizza and I feel so much better. [19:32:15] lol [19:32:17] :) [19:32:26] I'm sure the pizza didn't mind. [19:32:42] I was having some GI issues and considering skipping lunch because of upset stomach, but I was being silly and needed food. [19:32:43] ^_^ [19:36:15] hey folks. I was trying to use the revscoring 'extract' utility, and this is what I got: {"cache": "fCGJT04P{M0001HWnpx6.. and other weird characters", "rev_id": 18870298, "label": "False"}. [19:36:15] Is that "cache" an expected result? I thought that "extract" will add the feature values of rev_id based on the that I passed to the argument. [19:36:18] maybe halfak knows [19:36:48] yup. the cache contains the values you want [19:36:59] cv_train knows how to work with that [19:37:10] See all the makefiles for how the files are used. [19:38:49] I was wondering what format that cache is in? [19:39:45] halfak: alright. I am now reading the makefile [19:40:11] yeah. I wonder why using that "cache" instead of passing the real feature values [19:40:24] awight, it's base64 encoded pickle. That's sad because most stuff in the cache is JSON-able. But some of it isn't. [19:40:43] So we needed to pickle and then base64 to make it safe for JSON [19:41:09] glorian_wd, lots of reasons! We e.g. we can extract the features for all models and put them in the same cache. [19:41:24] We can also grow caches by running extract on observations that already have some cache. [19:41:26] darn. Yeah I was idly thinking that pure text might compress better. [19:42:07] cache is an awesome, flexible notion for extraction/dependency solving. [19:42:19] oh I see [19:42:20] kk [19:43:13] We used to just extract vectors of features and cache is superior in a lot of ways. [19:44:34] halfak: yeah I saw in the old edit quality tutorial, you used TSV as input file [19:44:47] Amir1: I made some changes to https://wikitech.wikimedia.org/wiki/Hiera:Ores and Hiera:Ores-staging, to remove the deprecated aptly repo. Do you know if that gets pushed to puppet automagically? [19:44:56] and according to the tutorial, it will generate an output TSV with real feature values [19:45:07] awight, I know that it'll get picked up in the next puppet run [19:45:12] great ty [19:45:26] which one is that glorian_wd ? [19:45:32] Can you file a task for updating it [19:45:32] ? [19:46:00] halfak: https://github.com/wiki-ai/editquality/blob/master/ipython/reverted_detection_demo.ipynb [19:46:13] that's an old edit quality tutorial [19:46:38] and also this: https://pythonhosted.org/revscoring/revscoring.utilities.html? [19:46:39] lol awight i thought https://docs.icinga.com/icinga2/latest/doc/module/icinga2/toc#!/icinga2/latest/doc/module/icinga2/chapter/monitoring-basics#alert-notifications was notifications.config but nope its templates.config [19:46:40] heh [19:46:50] They were outdated in so many ways :P [19:47:13] .config = conf [19:47:19] awight: you can push it with sudo puppet agent -tv [19:47:28] paladox: Developers get kind of freaky when they've been locked up on one project for a decade [19:47:32] Amir1: ty! [19:47:42] oh :) [19:48:10] awight: https://gerrit.wikimedia.org/r/#/c/361939/ :) [19:48:20] halfak: which links that you want me to put into the task? [19:48:53] 10Scoring-platform-team-Backlog: Remove custom apt repo from ores labs boxes - https://phabricator.wikimedia.org/T169129#3388332 (10awight) Should be done, I'll close this task once verified. https://wikitech.wikimedia.org/w/index.php?title=Hiera%3AOres&type=revision&diff=1763193&oldid=1761366 https://wikitech.... [19:49:46] glorian_wd, I don't see what's wrong with reverted_detection_demo.ipynb [19:50:37] halfak: in that book, the input file is TSV. AFAIK, the existing revscoring uses JSON [19:50:51] thanks :) [19:51:24] glorian_wd, the notebook writes its own output file [19:51:28] It can write whatever it wants. [19:52:48] halfak: can revscoring extract read a TSV file? [19:53:25] can 'revscoring extract' read a TSV file? [19:55:08] I am not talking about writing the output, but I am talking about reading the input file. [19:56:01] glorian_wd, it cannot [19:56:18] awight take 2, i will remove my ack [19:56:21] from the puppet failures [19:56:39] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 13 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:56:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:56:46] \o/ [19:56:48] here we go [19:57:37] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 14 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:57:40] Which is the *-quality repo for article quality? [19:57:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:57:44] halfak: yeah. In the book, all of the uploaded datasets are in TSV. For instance, /datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2 [19:57:52] hmm, it's not working awight ^^ [19:57:55] hmm [19:58:02] yeah plus my puppet fix is failing [19:58:13] k I'll see if I can resolve the puppet thing manually [19:58:35] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 15 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [19:58:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [19:59:05] ACKNOWLEDGEMENT - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] paladox Ack [19:59:11] ACKNOWLEDGEMENT - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 15 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] paladox Ack [20:00:04] awight it's called "wikiclass" [20:00:12] I'd be happy to move it to "articlequality" :))) [20:00:36] haha k I'll live with it for now [20:05:35] 10Scoring-platform-team-Backlog: Remove custom apt repo from ores labs boxes - https://phabricator.wikimedia.org/T169129#3388368 (10awight) 05Open>03Resolved a:03awight That worked, but didn't solve the apt problem I was hoping to fix. [20:07:00] harr. and --folds 1 failed three hours into the job, like halfak warned. [20:08:00] damn. Sorry awight. My new version will solve for this. Working on that PR now [20:08:05] No good for you though [20:08:08] What was the error? [20:09:31] damn [20:09:33] volite [20:09:35] halfak: https://phabricator.wikimedia.org/P5646 [20:09:36] i have enabled [20:09:40] * paladox disables it [20:10:08] halfak: Has docs on how to use "train" ? [20:10:14] damn. Protip if you want to run tests, do a tiny shuf before cv_train [20:10:22] revscoring train_model -h [20:10:57] hmm wait I've been trying to ask you about that shuf [20:11:08] Can't we skip it on the 833k set for draftquality? [20:12:51] lets try this again [20:12:52] take 3 [20:12:52] https://gerrit.wikimedia.org/r/#/c/361948/ [20:13:46] awight, we can, but the training process will be faster with a small shuf [20:14:00] I'd just write a data file with a 10k shuff for testing now [20:14:03] And then pipe that in [20:14:07] ? [20:14:27] I think codezee already ran a small set, and wanted me to verify using the full set [20:14:27] ? [20:14:39] awight, right but right now you're working out a proper workflow [20:14:50] And you just had to wait 30 mins for an error [20:14:58] yaya it worked awight [20:15:01] kk ty [20:15:08] note to self never enable volite [20:15:14] i mean [20:15:14] volatile [20:15:24] AH [20:15:31] weird, I don't fully grok [20:15:40] I read the docs but not sure how it applied here. [20:15:44] i found the answer here https://lists.icinga.org/pipermail/icinga-users/2016-November/011519.html [20:16:25] ooh [20:16:36] cat | shuf -n 10k > my_test_file.json [20:16:50] cat my_test_json | revscoring train_model ... [20:16:52] paladox: possibly premature high five! [20:17:02] halfak: gotcha, thanks [20:17:08] Thanks :) [20:17:08] haha [20:21:08] * halfak curses at pickle [20:34:00] awight|eat, please move tasks like https://phabricator.wikimedia.org/T169129 to the "Done" column on the main board so we know to report it! [20:35:37] 10Scoring-platform-team: Remove custom apt repo from ores labs boxes - https://phabricator.wikimedia.org/T169129#3388465 (10awight) [20:36:54] 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: Add common set of statistics to all threshold-based test-statistics - https://phabricator.wikimedia.org/T162150#3388490 (10awight) [20:37:38] 10Scoring-platform-team, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban): deployment-sca03 can't call puppetmaster - https://phabricator.wikimedia.org/T143958#3388509 (10awight) [20:38:36] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 25 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [20:38:41] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 17 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [21:16:48] halfak coincidence i had pizza today for dinner heh [21:16:50] dominos :) [21:17:49] * glorian_wd so much pizzas in this channel today :P [21:17:52] halfak: back to train_model, you were saying that I need to snip the training and set data manually. What do I do with the test data? I assume that isn't included in train_model? [21:18:05] * awight snifs around for leftovers [21:18:17] glorian_wd lol, i have it down the road from me :) [21:18:17] * awight sniffs at English spelling [21:18:21] lol [21:18:30] english spelling? [21:18:41] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [21:18:46] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 28 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ldap-utils] [21:19:07] paladox, :) [21:19:11] awight, test_model! [21:19:12] * awight lends paladox a germanic Capital [21:19:15] awight: maybe I can help answering your question. I think there's a utility for manually testing the model with test set [21:19:19] lol [21:19:23] halfak: lol these sneaky command names [21:19:27] :) [21:19:28] yes that's the one that was mentioned by halfak . test_model [21:19:45] :) [21:19:59] (train_model + test_model) * folds = cv_train [21:20:13] (in a basic sense) [21:20:19] i live around alot of fast food places heh [21:20:24] halfak: then why "extract"? why not extract_cache or extract_feature? :P [21:20:32] where dominos is that use to be buger king [21:20:49] used to be called "extract_features" but now "extract" works for any dependency set [21:22:12] * glorian_wd googled for 24h pizza [21:22:26] lol, it's open late where i am [21:22:46] i have around three mcdonalds possibly more all in driving distance (close by) [21:22:54] one of them i have never been too. [21:26:42] well they now have the tablets out. and on the + side mcdonalds starts trials at deliverying in the uk [21:26:44] wohoo lol [21:35:43] halfak: train_model doesn't take any of the "-s" arguments. I guess those are only for testing? [21:36:05] Right [21:36:11] Only test_model will take those [21:36:13] kk [21:36:31] btw ".DELETE_ON_ERROR:" was a huge breakthrough for my personal sanity. [21:36:45] what's that? [21:37:08] It causes makefiles to delete the target file if the commandline fails, rather than writing a zero-byte monstrosity. [21:37:21] OMG [21:37:23] OMGOMG [21:37:25] :D [21:40:13] awight did remove the old apt repo work? [21:40:14] fro ores-worker-07 [21:42:10] paladox: no, it didn't. [21:42:23] ok [21:42:36] awight does [21:42:39] apt-get clean cache [21:42:42] apt-get update [21:42:44] fix it? [21:42:50] no I tried that too. [21:42:55] ok [21:42:56] It was indeed cruft, though, so good riddance. [21:43:02] hahaha :) [21:43:04] I'm asking andrewbogott about the ldap issue, he seems to remember something like that [21:43:11] ok [21:43:13] thanks :) [21:44:07] It can be fixed by hand, I just don't like it. [21:44:19] oh [21:44:49] with the 40m notification interval, I feel okay leaving the problem until we can fix properly. Thanks again! [21:45:07] Ok and your welcome :) [21:47:12] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [21:47:26] yay [21:47:46] it recovered for ores-worker-07 [21:48:06] ores-web-04 fails with [21:48:07] CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [21:50:55] awight george washington [21:52:30] haha: father of ruthless real estate speculation. [21:56:15] who is "father of ruthless real estate speculation."? [21:58:41] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ores/99-main.yaml] [21:58:51] halfak: in revscoring "tune", I see an argument called