[11:12:20] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for nowiki - https://phabricator.wikimedia.org/T188235#4000743 (10jeblad) [11:50:02] wiki-ai/editquality#137 (fix_merge_labels_part_ii - 0316b2a : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/346243975 [12:47:18] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 52 seconds ago with 12 failures. Failed resources (up to 3 shown): Package[ldap-utils],Package[libnss-ldap],Service[nscd],Service[nslcd] [13:03:08] tgr|away: I just saw your message. It's complex it might be that the proper data hasn't pulled in [13:03:13] Let me check [13:05:51] akosiaris: Looks like you slayed ORES on scb*! [13:07:07] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for nowiki - https://phabricator.wikimedia.org/T188235#4000743 (10Ladsgroup) I don't understand the point of this phabricator task. Norwegian has list of bad words: https://github.com/wiki-a... [13:07:51] awight: yup, that's done :-) [13:08:15] akosiaris: Thanks! I’ll make a patch to reinstate the previous worker count, and we can see where that gets us... [13:08:15] awight: good morning and happy monday, feeling to review some stuff? :D [13:08:20] Amir1: yessir! [13:08:27] akosiaris: kudos! [13:08:55] awight: https://gerrit.wikimedia.org/r/#/c/413259/ [13:08:56] :D [13:09:57] (03CR) 10Awight: [C: 032] Add tests for new system of $wgOresModels [extensions/ORES] - 10https://gerrit.wikimedia.org/r/413259 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [13:10:32] https://github.com/wiki-ai/editquality/pull/134 [13:10:41] This is rather new :D [13:10:50] awight: is there anything I can take a look [13:11:18] Amir1: I’m imagining I’ll deploy Extension:JADE to the beta cluster today… [13:11:43] No patches outstanding. [13:11:50] cool [13:11:56] But you can help smoke test, if that sounds interesting? [13:12:29] (03Merged) 10jenkins-bot: Add tests for new system of $wgOresModels [extensions/ORES] - 10https://gerrit.wikimedia.org/r/413259 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [13:12:45] sure [13:13:28] (03CR) 10jenkins-bot: Add tests for new system of $wgOresModels [extensions/ORES] - 10https://gerrit.wikimedia.org/r/413259 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [13:15:38] Amir1: Ah, this is dramatically simpler and ready for re-review, https://gerrit.wikimedia.org/r/#/c/407467/ [13:16:53] awight: oh that's simple, just one thing. I thought we have it in gerrit [13:17:02] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:17:15] What’s the one thing? [13:17:29] https://gerrit.wikimedia.org/r/#/c/407467/5/puppet/modules/jade/manifests/extension.pp [13:17:44] the github mirror, I think we have jade ext. in gerrit now [13:17:50] haha thanks for noticing [13:18:05] :D [13:19:08] awight: tell me when you update it and I'll merge it [13:20:12] Amir1: Do you happen to know if we “needs_update” when the extension adds a namespace? [13:20:21] or is that just for schema changes? [13:25:05] I’m getting empirical on it: rebuilding the VM to see what happens. [13:26:49] Is there any research document available for ORES. I wanted to read about it, if it is publicly available [13:30:35] Guest19693: Good question! There are, but it seems hard to navigate to find them. Here’s the best resource I know of, https://www.mediawiki.org/wiki/ORES/Scholarship [13:34:45] Thank you for that. Is there anyway we can know the mechanism and how accurately the AI predicts? [13:35:23] awight: it's just for schema changes, the namespace stuff gets there automagically \o/ [13:35:37] Amir1: right on. Also, confirmed empirically :) [13:35:42] The new patch is pushed. [13:36:03] Sorry, was afk :D [13:37:28] Guest19693: The accuracy and other statistics are actually available dynamically, for example: https://ores.wikimedia.org/v3/ (for API docs) [13:37:30] Guest19693: https://ores.wikimedia.org/v3/scores/enwiki?models=damaging&model_info=statistics [13:39:23] Thats neat. So all of this is for the version 3 model trained on English wiki data ? [13:39:26] Guest19693: by “mechanism”, are you looking for information about how the models are trained? [13:39:49] Guest19693: Yes, the “v3” route is actually an API version and not the model version, though. [13:41:23] By mechanism I mean, the workflow, on how they get the data, any pre-processing of the data, architecture of the model used for training (like in deep learning we have different layers) and then the metrics, (which I guess are in the last link you shared) [13:44:08] Guest19693: cool. There are several repositories, one for each type of model, but for example here is where we define the workflow for “editquality” models, i.e. “damaging” and “goodfaith”: https://github.com/wiki-ai/editquality/blob/master/Makefile [13:44:42] I think everything you want is on display there. https://github.com/wiki-ai/editquality/blob/master/Makefile#L170-L185 [13:44:56] Guest19693: Can I ask what your project is? [13:45:27] I am actually doing a study on Wikipedia article quality assessment. [13:46:07] I came across two in particular ORES and another one which uses deep learning to assess the quality. [13:46:34] fun! Here’s the repository for article quality models, https://github.com/wiki-ai/wikiclass [13:47:17] Guest19693: Our research scientist, halfak, should be in this IRC channel in a few minutes. Stick around to say hi! [13:47:53] Great! Thanks. You saved me a lot of time [13:49:30] :D [13:50:07] Glad you jumped in to ask questions. These might be helpful as well, https://www.mediawiki.org/wiki/ORES/FAQ [13:50:54] Thanks! [13:56:17] akosiaris: I’m trying to get rid of the multiple defaults in puppet, but stuck trying to figure out where we’ll read the hieradata from. Seems that no file exists yet? [13:57:04] which key though ? [13:57:07] akosiaris: or…. I think I found it, hieradata/role/common/ores.yaml [13:57:30] akosiaris: for example, profile::ores::celery::workers [13:57:39] looks like I would put in that common/ores.yaml file? [13:58:21] yup [13:58:38] :) [14:04:10] akosiaris: My amateur butchery: https://gerrit.wikimedia.org/r/#/c/414666/ [14:04:23] argh, lint [14:05:12] hmm. I [14:05:23] I’m not sure what to do about the class parameter defaults [14:07:14] grr. [14:09:59] akosiaris: What do you think, should I just savagely leave these as “optional” parameters, which will break everything if they’re missing from hiera? [14:10:51] These are used in labs right ? I would leave them as is [14:11:02] ok [14:11:03] or else labs will probably break [14:11:50] /o\ [14:22:21] akosiaris: PS2 is ready to review. Also, I’m trying to adjust the Grafana queries, and wondering if there’s anything I can do to drop the scb* nodes from “ores.*” ? [14:23:02] I think those dashboard are hostname based [14:23:16] so there should be a single line matching that [14:23:58] e.g. https://grafana-admin.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=10&fullscreen&edit&orgId=1 [14:24:21] it's enough to drop K, A and B-J series I think [14:24:58] https://grafana-admin.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=3&fullscreen&edit&orgId=1 is way easier. It's just B,C that need to be dropped [14:25:01] doing so now [14:25:23] I’m fine with that, it’s just cleanup. “ores.*” used to refer to ores.scb* and ores.ores*, so I was hoping that ores.* would only refer to the ores* nodes at some point. [14:25:54] I’m assuming this is simply because ores.scb* data is sitting around in the sources. [14:26:05] yeah it's going to be there for some time [14:26:13] but it will be cleaned up eventually [14:26:18] perfect [14:26:28] don’t worry about the busywork, I’m already part-way through [14:26:55] ok [14:28:43] Looks like celery died on ores1008 [14:30:26] celery processes are running though [14:30:32] and I don't see anything in the logs [14:31:17] if it's really dead it hasn't logged a single thing since Feb 21 [14:32:21] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for nowiki - https://phabricator.wikimedia.org/T188235#4001339 (10jeblad) This is not listed at [[ https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists | m:Resea... [14:34:20] Is it correct to assume that in the edit quality support (https://www.mediawiki.org/wiki/ORES#Basic_support) the BWDS is used to predict whether an article will be reverted or not? [14:34:33] wiki-ai/revscoring#1429 (w2v - 19888ac : Sumit Asthana): The build was broken. https://travis-ci.org/wiki-ai/revscoring/builds/346307261 [14:35:42] o/ Guest19693 [14:35:44] Heard you were looking into article quality models. [14:36:34] Yes sir, I was. I found it really interesting for a study I am doing on article quality assessment. [14:41:51] Guest19693, cool. Let me know how I can help [14:42:09] Hey halfak [14:42:23] o/ Vemont [14:43:22] *r [14:43:50] hi [14:44:55] I am still going through the documentation (the announcement and the wiki for ORES) so far. Is there a workflow documentation for the data pre-processing and architecture of the models used? [14:45:32] Guest19693: regrettably, there isn’t any documentation for the workflows. [14:45:46] The model architecture is essentially sklearn’s GradientBoosting [14:45:47] wiki-ai/revscoring#1431 (w2v - 9d2a59a : Sumit Asthana): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/346312216 [14:47:23] Okay. I think I will go through the documentation and all the resources that I have had till now and come back with questions in a day or two. Thank you everyone! [15:34:19] 10Scoring-platform-team (Current), 10JADE, 10WMF-Communications: Blog about JADE - https://phabricator.wikimedia.org/T183200#4001554 (10awight) a:03awight [15:48:26] 10Scoring-platform-team, 10Global-Collaboration, 10MediaWiki-extensions-ORES: [Epic] Audit of pending ORES GUI deployments - https://phabricator.wikimedia.org/T187836#4001631 (10Halfak) [15:49:15] 10Scoring-platform-team (Current), 10JADE: Deploy JADE prototype in Beta Cluster - https://phabricator.wikimedia.org/T176333#4001634 (10awight) a:03awight [15:51:28] 10Scoring-platform-team (Current), 10ORES: Investigate 3x scoring load after ORES changeprop migrated to new precache endpoint - https://phabricator.wikimedia.org/T187928#4001652 (10Halfak) a:05Pchelolo>03awight [15:53:03] 10Scoring-platform-team, 10User-Ladsgroup: Train/test damaging/goodfaith model for Catalan Wikipedia - https://phabricator.wikimedia.org/T187732#4001656 (10Halfak) [15:54:13] 10Scoring-platform-team (Current), 10User-Ladsgroup: Train/test damaging/goodfaith model for Catalan Wikipedia - https://phabricator.wikimedia.org/T187732#3983664 (10Halfak) [15:55:26] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10MediaWiki-extensions-ORES, 10Regression: ORES extension highlights edits that are patrolled - https://phabricator.wikimedia.org/T187337#4001660 (10Halfak) [15:56:26] 10Scoring-platform-team (Current): Support for word2vec on ORES deployment - https://phabricator.wikimedia.org/T187217#4001676 (10awight) [15:57:18] 10Scoring-platform-team (Current): Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#3968196 (10awight) [15:57:39] 10Scoring-platform-team, 10drafttopic-modeling: Refactor scripts fetching text and other metadata - https://phabricator.wikimedia.org/T181074#4001680 (10Halfak) [16:19:25] halfak: tell me when you have some time to talk about wrapping up the work [16:21:28] Amir1, now is good. :) [16:22:11] Sure thing [16:22:35] Want to voip? [16:22:36] halfak: the work is in two part: Makefile automation and the extension support for wp10 model [16:22:53] now at the dev room, need to find a quiet space [16:22:58] will be there in one minute [16:23:23] kk call when ready [16:25:01] halfak: invited you [16:41:04] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Simplify hewiki, cswiki, plwiki, svwiki - https://phabricator.wikimedia.org/T188270#4001936 (10Halfak) [16:41:35] 10Scoring-platform-team (Current): Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#3968196 (10awight) Working on the Debian packaging here: https://phabricator.wikimedia.org/source/word2vec/ @Sumit Is the gensim package able to read the gzipped file, or should we decompre... [16:42:17] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Re. init enwiktionary reverted model & labeling campaign - https://phabricator.wikimedia.org/T188271#4001950 (10Halfak) [16:42:43] 10Scoring-platform-team (Current), 10User-Ladsgroup: Fix or decide on edge cases of Makefile - https://phabricator.wikimedia.org/T186453#3944484 (10Halfak) he, cs, pl, sv -- all are the weird mix of needs_review and not. This can be cleaned up into a single templated form. (T188270) enwiktionary -- the lab... [16:45:47] 10Scoring-platform-team (Current), 10User-Ladsgroup: Fix or decide on edge cases of Makefile - https://phabricator.wikimedia.org/T186453#4001973 (10Halfak) Wikidata just has a checked-in file to kick it off. See if we can work around that. [16:46:10] 10Scoring-platform-team (Current), 10User-Ladsgroup: Fix or decide on edge cases of Makefile - https://phabricator.wikimedia.org/T186453#4001978 (10Halfak) @Ladsgroup said he would double-check Hungarian. [17:06:48] 10Scoring-platform-team (Current): Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4002140 (10awight) Here's what the package will look like installed, for now: ``` dpkg -L word2vec /. /usr /usr/share /usr/share/doc /usr/share/doc/word2vec /usr/share/doc/word2vec/changelog... [17:07:33] 10Scoring-platform-team (Current): Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#3968196 (10awight) a:03awight [17:10:02] 10Scoring-platform-team, 10ORES, 10Documentation: Write an ORES manual - https://phabricator.wikimedia.org/T188280#4002177 (10Halfak) [17:10:53] halfak: ^ IMO we were leaning towards consensus that the manual would be aimed at tool devs, but that looks more like a scoring-platform-internal manual? [17:11:28] Na. Was thinking it would be primarily for tool devs and secondarily for development/maintenance. [17:13:41] Does seem like very different audiences... [17:19:23] 10Scoring-platform-team, 10Release-Engineering-Team: Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281#4002206 (10awight) [17:22:47] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 2 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4002231 (10Halfak) [17:24:13] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 2 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4002239 (10MarcoAurelio) Would this enable the nice backgrounds on Special:RecentCh... [17:24:57] 10Scoring-platform-team, 10monitoring: Create Iginca alerts for dead celery workers - https://phabricator.wikimedia.org/T188282#4002247 (10awight) [17:29:31] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 2 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#2131547 (10jmatazzoni) >>! In T130279#4002239, @MarcoAurelio wrote: > Would this en... [17:30:56] 10Scoring-platform-team (Current), 10Beta-Cluster-Infrastructure, 10Recommendation-API, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): What to do with deployment-sca03? - https://phabricator.wikimedia.org/T184501#4002283 (10greg) [17:31:17] 10Scoring-platform-team (Current), 10MediaWiki-Core-Tests, 10MediaWiki-extensions-ORES, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), and 2 others: How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#4002285 (10greg) [17:32:06] 10Scoring-platform-team, 10JADE: Where should we host permalinks to JADE JSON schemas? - https://phabricator.wikimedia.org/T188284#4002302 (10awight) [17:36:30] 10Scoring-platform-team, 10Mobile-Content-Service, 10ORES, 10Operations, and 2 others: Limit resources used by ORES - https://phabricator.wikimedia.org/T146664#4002357 (10awight) 05Open>03Resolved a:03awight We've moved to a dedicated cluster—the best possible way to limit resources ;-) [17:46:10] 10Scoring-platform-team, 10ORES, 10Performance: Profile ORES code memory use, optimize pre-fork allocation - https://phabricator.wikimedia.org/T182350#4002410 (10awight) [17:49:32] halfak: srrodlund wants to know if we can skip the docs meeting today. WFM. Anything urgent you wanted to talk about? [17:49:46] na. It's cool [17:51:47] Sweet! Thanks folks! [17:53:50] halfak: dang, forgot to mention that I need to take Wednesday off for moving. [17:54:00] Adding to the calendar now. I’ll try to catch up on Friday. [17:54:10] OK thanks for letting us know :) [17:54:32] Could you go add some lines about your OoO coming up to the staff etherpad? [17:54:34] awight, ^ [17:54:40] +1 [17:55:08] hehe I’ll miss that meeting specifically. [18:17:26] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4002614 (10awight) [18:17:34] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#3968196 (10awight) @akosiaris Feel like reviewing this debian packaging for sanity? [18:23:32] Lunch! [18:23:47] When I get back, I'm going on a reviewing spree [18:25:41] * awight quickly resumes marching up and down the square [18:42:29] 10Scoring-platform-team (Current), 10ChangeProp, 10ORES: Drop "non bot" condition from ORES changeprop rules - https://phabricator.wikimedia.org/T187927#4002754 (10awight) Our team still needs to discuss the details. [19:06:56] (03PS9) 10Awight: [DNM] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) [19:28:52] 10Scoring-platform-team, 10Scap: Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281#4003007 (10greg) Using a fan out method is how this is handled normally. [19:29:44] awight: halfak|Lunch https://github.com/wiki-ai/editquality/pull/135 [19:32:50] wiki-ai/editquality#140 (template_huwiki - a65ceff : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/346444948 [19:33:11] 10Scoring-platform-team, 10Scap: Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281#4003034 (10awight) >>! In T188281#4003007, @greg wrote: > Using a fan out method is how this is handled normally. @greg Sorry, I'm not sure how to parse that. I think that's co... [19:37:29] That was fast, thank you! [19:37:56] 10Scoring-platform-team (Current), 10User-Ladsgroup: Fix or decide on edge cases of Makefile - https://phabricator.wikimedia.org/T186453#4003052 (10Ladsgroup) [19:38:50] awight: For the first time since 2015, the Manual Makefile is less than 1000 line [19:39:08] Amir1: You’re scattering it to the wind! [19:39:20] BTW, Ext:JADE is about to go out with SWAT [19:45:45] \o/ [20:01:40] awight: halfak I made an intersting discovery, we have not just reverted but goodfaith and damaging model even though the labeling campaign (mentioned in the Makefile) has only three labels [20:01:55] in English Wiktionary [20:02:03] ouch [20:02:10] We really need to go through our stuff [20:02:31] It’s as if we’re understaffed or something [20:03:49] Woah. [20:03:52] lol [20:04:01] Damn. Good detective work! [20:09:58] JADE didn’t make into SWAT. I’m seeing if I can deploy myself, but probably not cos I need to leave soon for a couple of hours. [20:12:57] In beta? Do you need a swat for beta? [20:14:09] well. It involves mediawiki-config files which are shared with production [20:14:31] Should be a no-op for production, but I wanted to be polite. [20:16:44] awight: halfak ^ [20:17:05] Oh! makes sense [20:17:28] wiki-ai/editquality#143 (enwiktionary_templating - e929744 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/346466007 [20:18:32] I haven't checked out the file as I wanted to do it in another PR (using stats machine) [20:21:13] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for nowiki - https://phabricator.wikimedia.org/T188235#4003260 (10Ladsgroup) This is not how revscoring works. Bad words are being used as proportion of added or removed, so for example usin... [20:24:30] 10Scoring-platform-team (Current), 10JADE, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Extension review for JADE - https://phabricator.wikimedia.org/T188307#4003278 (10awight) [20:25:25] halfak: enwiktionary is up, wikidata seems to be more edge-casy than I thought :( [20:25:45] 10Scoring-platform-team (Current), 10JADE, 10Security-Reviews: Security review for Extension:JADE - https://phabricator.wikimedia.org/T188308#4003289 (10awight) [20:25:49] one thing for example, tuning report does have pop-rate but model part doesn't [20:25:51] Saw that. Working through the word2vec PR right now. Will be on that next [20:26:04] Amir1, oh yeah. We should drop the pop rate in tuning. [20:26:17] awight, wooo for security review! [20:26:37] Luckily, legoktm caught me red-handed... [20:26:41] * awight wrings gloves [20:26:51] I'm leaving for the day, already worked for ten hours today, if that's goes on this way, I never be able to reduce my overtime [20:27:05] Amir1: you gotta chill :p [20:27:40] 10Scoring-platform-team (Current), 10JADE, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Extension review for JADE - https://phabricator.wikimedia.org/T188307#4003327 (10awight) [20:27:47] I'd love to but it seems Wiki doesn't like that idea ;) [20:29:17] 10Scoring-platform-team (Current), 10JADE, 10Patch-For-Review: Deploy JADE prototype in Beta Cluster - https://phabricator.wikimedia.org/T176333#4003348 (10awight) [20:29:19] 10Scoring-platform-team (Current), 10JADE, 10Security-Reviews: Security review for Extension:JADE - https://phabricator.wikimedia.org/T188308#4003289 (10awight) [20:31:51] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Re. init enwiktionary reverted model & labeling campaign - https://phabricator.wikimedia.org/T188271#4003364 (10Ladsgroup) https://github.com/wiki-ai/editquality/pull/136 [20:33:38] o/ Amir1 [20:33:42] have a good evening :) [20:33:55] I'm still here, cleaning up documents :D [20:34:09] I need to get an uncharacteristic massage. [20:34:15] relaxing: gross. [20:37:23] 10Scoring-platform-team, 10Beta-Cluster-Infrastructure, 10ORES: Beta: Could not find class role::ores::worker - https://phabricator.wikimedia.org/T188316#4003417 (10thcipriani) [20:38:39] k, biab [20:49:06] I'm leaving for the day now [20:49:09] see you soon [20:49:16] o/ [20:52:32] o/ [20:54:42] Just coughed some cookie up the back of my nose. It's been 5 minutes and I'm still recovering. [20:54:45] OMG. [20:54:50] how did this happen [21:19:46] I'm trying to work on the ORES deployments we discussed on Friday, but nothing works in beta labs because ores-beta.wmflabs.org keeps 503ing on all but the most simple requests [21:19:59] example request? [21:20:13] https://ores-beta.wmflabs.org/v3/scores/ returns a list of wikis but https://ores-beta.wmflabs.org/v3/scores/enwiki/ returns "server overloaded" [21:20:36] The actual request MW is trying to make is https://ores-beta.wmflabs.org/v3/scores/simplewiki/?models=damaging&model_info=statistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.99%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.15%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.45%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.9%22&format=json [21:23:40] RoanKattouw, I bet that celery is down for some reason. Let me see if I can kick it. [21:24:25] Hmm.. Looks like I can't connect to sca03. [21:24:32] I think we moved to a new host recently. [21:24:35] Docs are out of date. [21:24:55] * halfak checks scap config [21:25:25] aha [21:25:26] deployment-ores01.deployment-prep.eqiad.wmflabs [21:25:59] indeed celery is offline [21:26:59] hmm.. I don't see a celery service available to even restart at all. [21:28:48] (03CR) 10Thcipriani: "I noticed that scap/scripts/promote_check.sh was added with permissions 644." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [21:28:52] RoanKattouw, I don't suppose you're in the deployment prep project in cloud vps are you? [21:30:54] I am [21:31:00] What do you need? [21:35:25] ( halfak ) [21:35:50] RoanKattouw, I'm trying to figure out what puppet classes are enabled on that host. [21:36:01] I can look that up in Horizon if you want [21:36:05] Thanks :) [21:36:10] Also cat /etc/motd might tell you some things [21:36:18] Oh! I am in deployment-prop [21:36:39] role::ores::worker ... hmmm [21:38:11] OK got it. [21:38:25] For some reason the celery service wasn't listed with the others so I needed to remember the exact name. [21:39:53] RoanKattouw, it's back [21:39:57] Sorry for the trouble! [21:40:16] Yay [21:40:16] Can confirm [21:41:42] \o/ [21:43:03] Threshold fetching is working in eval.php now, but I'm not seeing the ORES filters on Recentchanges yet, investigating that now [21:50:24] D'oh, I'm eval-ing on simplewiki but I had testwiki open in my browser [21:51:55] :P [22:03:23] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: OneVsRest Classification for revscoring - https://phabricator.wikimedia.org/T185896#3927992 (10Halfak) Merged! [22:03:28] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: OneVsRest Classification for revscoring - https://phabricator.wikimedia.org/T185896#4003713 (10Halfak) 05Open>03Resolved [22:06:32] wiki-ai/revscoring#1433 (master - 98f0951 : Aaron Halfaker): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/346514862 [22:49:25] back. [22:56:05] (03PS10) 10Awight: [DNM] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) [22:56:15] o/ awight [22:56:46] I'm just about to step away. Wanted to let you know that Roa.n is working on getting those 4 wikis running the ORES filters now. [22:56:59] I just started the celery service on beta because it was down :| [22:57:01] rad, I’ll be around for 2 hours. [22:57:03] Didn't have time to investigate. [22:57:20] halfak: ah, thcipriani mentioned that puppet was failing, and created a task for us. [22:57:34] Nice. [22:58:42] I’m finally able to play with the cached venv patch on beta. [23:01:17] PROBLEM - puppet on ORES-worker03.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:01:38] O_O i think it worked. [23:01:56] PROBLEM - puppet on ORES-worker04.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:04:42] naw [23:04:43] close, though [23:09:28] (03PS11) 10Awight: [DNM] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) [23:12:35] PROBLEM - puppet on ORES-worker05.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:18:36] (03PS12) 10Awight: [DNM] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) [23:25:16] OK I'm off. [23:25:17] o/ [23:25:23] 10Scoring-platform-team, 10ORES: Use ORES "promote" check to actually check our services - https://phabricator.wikimedia.org/T188341#4004128 (10awight) [23:26:58] (03CR) 10Awight: "Okay, this patch is finally smoke tested and works as designed. Rollback doesn't spend any time rebuilding the venv," [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [23:27:44] 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3778593 (10awight) Do not merge patches, currently blocked on scap 3.8 dep... [23:28:10] see ya! [23:31:29] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 6 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4004174 (10Catrope) Alright! Sorry for the late notice, but I'll schedule this for the SWAT deployment at 4pm Paci... [23:31:46] RECOVERY - puppet on ORES-worker03.experimental is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:31:47] RECOVERY - puppet on ORES-worker04.experimental is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:34:36] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix - https://phabricator.wikimedia.org/T181010#4004181 (10awight) [23:34:41] 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: [Blocked] Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4004179 (10awight) 05Open>03stalled [23:41:03] RECOVERY - puppet on ORES-worker05.experimental is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures