[06:43:40] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert. [09:17:10] Lydia_WMDE: https://phabricator.wikimedia.org/T180694 [09:17:13] FYI [09:17:25] sorry, wrong channel :D [09:42:46] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting. [13:01:53] (03PS1) 10Ladsgroup: Move Contributions and ChangesList hook handlers to their dedicated files [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392415 [13:03:55] (03CR) 10jerkins-bot: [V: 04-1] Move Contributions and ChangesList hook handlers to their dedicated files [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392415 (owner: 10Ladsgroup) [13:12:00] (03PS2) 10Ladsgroup: Move Contributions and ChangesList hook handlers to their dedicated files [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392415 [13:39:42] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3774296 (10Aklapper) [15:33:44] o/ [15:34:16] o/ [15:37:05] Oh hey! I think my earlier messages didn’t go through... [15:37:13] Jump ball: https://gerrit.wikimedia.org/r/#/c/391562/ [15:37:25] Amir1 reviewed already on the labs repo, fyi [15:38:29] awight, what's the purpose of the pip_install make rule? [15:40:48] Oh I see that it does pip install pip --upgrade before anything. [15:43:09] halfak: the pip_install populates the virtualenv with the required packages [15:43:13] It’s implicit in the older version of the Makefile, but I broke it out into its own step. c.f. “frozen-requirements.txt” in https://github.com/wiki-ai/ores-wmflabs-deploy/blob/e81296f3099773ed900382928d5e8fdd6d3173f0/Makefile [15:43:13] PROBLEM - puppet on ores-worker-06 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:43:41] the —upgrade pip is just a detail, not the purpose of the rule. [15:44:00] awight, still important because installing the wheels won't work otherwise. [15:44:21] Actually, in this case we can remove that line [15:44:29] I learned more since writing this [15:44:39] Why loop through the requirements doc? [15:44:43] wheels are only forward-compatible [15:44:50] Why not pip install -r requirements.txt? [15:44:54] PROBLEM - puppet on ores-worker-08 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:44:57] cos there are multiple files [15:45:00] Oh... wait. I see. [15:45:02] yeah [15:45:05] and pip doesn’t allow -r F1 -r F2 [15:45:08] unfortunately. [15:45:31] Sorry if I seem async, something’s up with wifi at this cafe and I’m following along using the irc logs [15:46:14] lol it's working :) [15:46:29] I have cell tethering as backup! [15:46:35] real-time irc log monitoring :P [15:46:54] lol I need a bot to read the offline logs and SMS me [15:47:41] PROBLEM - puppet on ores-web-03 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:48:02] anyone know what's up with ^ ? [15:48:10] I just noticed there’s one more step we should add: purging the active venv. Seems a bit dangerous in case the user is not in venv, though. Will see if there’s an easy environment variable to gate on [15:48:21] no, but it looks like labs business [15:48:38] kk [15:50:09] halfak: following suit of wikiclass scripts, i've added an additional script to fetch text for articles so that dependency extraction can be done offline [15:50:11] (03CR) 10Halfak: [C: 04-1] Put all the wheel tools into the Makefile (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) (owner: 10Awight) [15:50:26] awight, just one note that we should install requirements for https://github.com/wiki-ai/draftquality/blob/master/requirements.txt [15:50:33] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:50:38] nice. [15:51:40] halfak ive noticed that etherpad i made some time ago hasnt been modfied yet just wanted to remind you, i realise your a pretty busy person so no worries (link: https://etherpad.wikimedia.org/p/ORES_FAQ_Email ) [15:53:12] awight & srodlund ^ [15:53:34] I was hoping they'd work towards the docs email because they'd work on the docs so much. [15:53:59] halfak: ok just wanted to say something. [15:55:17] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:56:02] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:57:50] FYI: chasemp broke puppet on cloud so these errors ^^^ should be fixed hopefully shortly [16:01:08] ACKNOWLEDGEMENT - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues zppix cloud services is fixing the issue. [16:06:02] halfak: awight: Hey [16:06:06] I missed your pings [16:06:31] I made this patch [16:06:32] https://gerrit.wikimedia.org/r/#/c/392415/ [16:09:01] It's practically moving the code around so I can make more space for T175757 and one other task [16:09:02] T175757: Store wp10 predictions in page_props table. - https://phabricator.wikimedia.org/T175757 [16:09:11] the extension needs a little bit of clean up [16:10:41] (03PS2) 10Awight: Put all the wheel tools into the Makefile [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) [16:11:27] 10Scoring-platform-team, 10Community-Tech, 10MediaWiki-extensions-ORES, 10MediaWiki-extensions-PageAssessments: Implement ORES wp10 predictions in PageAssessments tool - https://phabricator.wikimedia.org/T156820#3774805 (10Ladsgroup) T166427#3709763 will address this task as well. [16:12:40] eek that’s a big one [16:12:44] RECOVERY - puppet on ores-worker-06 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:13:13] I know :/ [16:14:33] RECOVERY - puppet on ores-worker-08 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:18:11] RECOVERY - puppet on ores-web-03 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:19:50] Thanks, this is a good change. [16:20:06] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:21:51] (03CR) 10Awight: [C: 032] "Thanks for taking the time to do this!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392415 (owner: 10Ladsgroup) [16:22:23] 10Scoring-platform-team (Current), 10ORES: ORES 500s when model_info lookup fails due to a key error - https://phabricator.wikimedia.org/T179712#3774834 (10Halfak) We're still waiting on getting this code deployed. I'm not clear what's the hold-up as I've been OOO for a while. @awight? [16:24:09] 10Scoring-platform-team (Current), 10ORES: ORES 500s when model_info lookup fails due to a key error - https://phabricator.wikimedia.org/T179712#3774839 (10awight) @Halfak No more holdups, rather I think the urgency decreased once frwiki config was fixed. [16:24:23] 10Scoring-platform-team (Current), 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Article quality campaign for Persian Wikipedia - https://phabricator.wikimedia.org/T174684#3774840 (10Ladsgroup) I did some analysis on distribution of page sizes over three groups: 1- [[https://quarr... [16:24:46] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:24:59] halfak: The worst of the issues was the surprise revscoring incompatibility once we removed the old modules that were still being referenced. Thanks for the ping, this is something I can do today. [16:25:12] cool :) [16:25:31] RECOVERY - puppet on ores-web-05 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:26:52] (03Merged) 10jenkins-bot: Move Contributions and ChangesList hook handlers to their dedicated files [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392415 (owner: 10Ladsgroup) [16:36:48] Amir1: we having some social time fyi [16:36:57] You are cordially invited :) [16:37:08] codezee: ^ if you wish! [16:37:13] yeah [16:37:19] I'm joining [16:43:00] Oi? [17:49:21] (03PS1) 10Petar.petkovic: [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [17:50:30] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [17:53:17] 10Scoring-platform-team (Current): Build mid-level WikiProject category training set - https://phabricator.wikimedia.org/T172321#3494777 (10Halfak) a:03Sumit [17:53:33] 10Scoring-platform-team (Current), 10Patch-For-Review: Clean up ORES wheels Makefile - https://phabricator.wikimedia.org/T180496#3775170 (10Halfak) a:03awight [17:54:10] I need to step away briefly. [17:54:14] back in a bit [18:03:23] awight: you were mentioning a way to inject features or sth for debugging, do you have it somewhere? [18:04:32] awight i got logstash working with gerrit :) [18:08:44] o/ [18:08:46] back [18:09:47] halfak: regarding the last item, till the time we don't have space for word2vec, can we use hashed words as features into a classifier or something like that ? [18:10:17] codezee: Hey what I was pointing out was just that there are existing vector features you can pry into. There is a way to inject additional features though, lemme get you an example: [18:11:13] awight: oh, i see i didn't quite get back then... [18:11:22] we *can* use hashed words in the meantime, yeah [18:11:47] We'll need to develop a workflow for selecting a subset of words to be included in our predictive features. [18:12:03] word/hashes [18:12:23] halfak: since same word in different docs will always generate the same hash, i don't think we require to build an vocab index first as is typically required for tfidf and stuff? [18:13:45] codezee: This is kind of an oblique example, but maybe chasing the call stack down it will help. https://github.com/wiki-ai/ores/blob/master/ores/utilities/test_api.py#L66 [18:13:56] There’s an example of injecting a specific feature value. [18:14:36] (disclaimer, I haven’t done this yet myself) I think it’s “injection_cache” that you want to play with. [18:17:31] halfak: i think its something like this we need - http://scikit-learn.org/stable/modules/feature_extraction.html#feature-hashing [18:19:29] codezee, right. note in the gist I shared with you that I experiment with that. however, it is too limited for us I believe. [18:19:51] codezee, we'd build the tfidf or selector on the hashes themselves. [18:20:06] * halfak finishes one reinbursement [18:23:41] (03PS2) 10Petar.petkovic: [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [18:24:38] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta cluster, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3775259 (10Etonkovidova) Checked in betalabs - the ORES filters are present. [18:24:45] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta cluster, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3775260 (10Etonkovidova) 05Open>03Resolved [18:25:18] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [18:25:43] 10Scoring-platform-team (Current), 10ORES: ORES 500s when model_info lookup fails due to a key error - https://phabricator.wikimedia.org/T179712#3775265 (10elukey) It would be great to have this code deployed soon, so we'll avoid false 5xx drills in the ops chan :) [18:26:28] halfak: i had a doubt though, does it matter if we use raw words as input to a selector(tfidf) or hashes bec even in case of a raw words, the algorithm will build a word->index mapping and use it internally [18:26:49] i think i'm missing the actual benefit of hashing... [18:31:26] (03PS1) 10Awight: Remove old revscoring 2.0.9 wheel [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/392459 [18:35:48] relocating [18:40:47] codezee, hashed words. Otherwise there's no purpose in hashing [18:41:38] halfak: i menat to ask you're suggesting hashing typically to tackle dimensionality? [18:41:47] right [18:43:37] halfak: and for selecting a subset we might need to filter out words commonly occuring everywhere, basically what tf-idf does, right? [18:43:58] codezee, YES [18:44:03] sorry capslock [18:45:16] halfak: last thing - revscoring presently has capability to take a single label, how can we move here? [18:45:44] where we have 30-40 odd categories and each example belonging to some of them [18:45:49] codezee, ? it does? [18:46:20] let me get a snapshot of dataset [18:46:38] codezee, no. What's the problem with revscoring? [18:47:06] halfak: oh, i was not aware of multiclass support [18:47:34] halfak: sorry i remember now wikiclass is inherently multiclass [18:48:04] so there i *can* look [18:49:10] halfak: but do you think having around 40 labels is doable ? [18:49:15] :) Not sure how RandomForest will take in multi-labels but it *should* work [18:49:20] yes [18:51:40] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy, 10Google-Code-in-2017: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#3775380 (10Aklapper) [18:52:50] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy, 10Google-Code-in-2017: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#2533822 (10Aklapper) [18:55:30] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy, 10Google-Code-in-2017: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#3775386 (10Aklapper) Imported as https://codein.withgoogle.com/tasks/5376624163815424/ [18:59:11] halfak: would supplying labels in an array format like - https://dpaste.de/9o6Z work out of the box? or some preprocessing might be required? [19:00:32] Not sure. I think it will work. [19:00:45] ok, i'll try [19:25:11] https://storia.me/pt/@pedrozambarda/inteligencia-artificial-apaga-farsas-4elktb [19:32:00] nice...the screens, terminals and code at the back in the picture give the impression of a hacker's workstation :D [19:38:09] halfak: i wanted to ask why would there be a need to supply True/False labels in tuple arguments to tfidf.fit - https://github.com/wiki-ai/revscoring/blob/master/revscoring/datasources/meta/tests/test_selectors.py#L19 ? [19:38:37] codezee, what? [19:38:52] What do you mean "why would there be a need to supply True/False labels in tuple arguments" [19:39:17] Oh! Like why supply labels at all? [19:39:24] yes [19:39:25] halfak: we don't need document labels while fitting, we just generate term frequencies and inv. doc freq. [19:39:32] conventionally i men [19:39:34] *mean [19:39:37] Because we're comparing tfidf within label to do selection [19:39:48] tfidf by itself is just weighting. [19:41:08] so scaling here is per label basis? and not across the entire set then? [19:42:12] scaling is not the goal [19:42:15] selection is [19:42:40] selection is based on the basic power of a term as a discriminator. [19:47:06] Changing locations. [19:47:10] Back online in 30 minutes. [19:49:50] oh, ok i get your point, ...i'll need to tweak this code a bit to accept multi-labels first of all to begin with [20:03:19] (03PS1) 10Awight: Require revscoring 2.0.10 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392474 [20:03:34] Amir1: are you around for that? ^ [20:04:01] I have more submodule nudging to do, but I think this one is enough to deploy safely. [20:04:14] (03CR) 10Ladsgroup: [V: 032 C: 032] Require revscoring 2.0.10 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392474 (owner: 10Awight) [20:04:23] awight: merged :) [20:04:57] O_O [20:04:59] ty! [20:05:08] that was quick :D [20:05:18] wiki-ai/editquality#9 (update_revscoring - e05db88 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/304926768 [20:06:18] wiki-ai/wikiclass#17 (update_revscoring - ec8986c : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/wikiclass/builds/304927316 [20:09:37] (03CR) 10Awight: [C: 032] Remove old revscoring 2.0.9 wheel [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/392459 (owner: 10Awight) [20:10:05] (03CR) 10Awight: [C: 032] "Self-merged lib cleanup" [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/392459 (owner: 10Awight) [20:13:17] * awight grinds teeth about some repos requiring manual merge and others breaking Gearman/Zuul on manual merge [20:18:36] I'm out [20:18:37] o/ [20:23:11] Amir1: good night! [20:23:16] o/ [20:23:48] codezee: Did you get any traction on the injection problem? [20:26:08] awight: i didn't take a look at it yet, trying to figure out selectors code and a potential bug rn...but thanks for the link, I'll have a look :) [20:26:22] kk, yeah you have plenty on your plate! [20:26:58] I’d be happy to try to rewrite the injection thing if I can be helpful, maybe tomorrow? [20:30:29] awight: rewrite as in? i think the link you gave it self explanatory, just add the feature value pair in request [20:30:34] *is self [20:30:59] oh cool, I hadn’t read enough of the rest of your scripts to see whether that would fit. [20:31:33] Also not sure what injecting a vector would look like, but if that doesn’t work I can tamper with the API endpoint to fix. [20:32:44] awight: yes i'm not sure how vectors fit in the whole scene bec i didn't find any live example using those [20:33:12] otherwise its cool [20:33:20] I can figure out which features are currently vectors… one moment please... [20:39:57] codezee, had a thought. I think tfidf selection is going to be a pain. [20:40:51] halfak: reason? [20:41:09] Instead, I think you should train a model on all of the text features (bag of words vector) and then use the feature_importance vector to select the top terms. [20:44:07] halfak: so individual features for a document should be raw freq counts or present/not present bools? [20:44:25] codezee, I think we'll get more signal out of frequency counts. [20:44:47] Maybe frequency divided by total document length [20:44:54] Avg frequency, if you will [20:45:03] But raw freq should be useful :) [20:48:10] yes freq / doc_len seems appropriate but for a comparison perspective i think it wouldn't harm to test it once with selection first then training too just to get and idea [20:48:11] halfak: Sorry, I thought you were gone for the afternoon—you still planning on doing the ORES docs meeting? [20:48:27] yes. [20:48:42] But if you want to skip it, I'm OK with that [20:48:42] I’m deploying so rescheduled with srrodlund. Lemme ask if she can still do today. [20:49:03] halfak: ^ your call! [20:49:11] Na. don't sweat it. Mostly, I want srrodlund to write an announcement email for the FAQ :) [20:49:11] Oh, I moved it to tomorrow [20:49:16] Can talk about that tomorrow. [20:49:20] Cool! [20:50:16] halfak: ok, so i'll first focus on getting freq counts tested, should be quick, will give us a rough idea where we stand with the dataset, and if we need any resampling [20:51:15] halfak: one reservation i had about using hashes is we'll never be able to say which words are useful for a particular category in a post-analysis step, we'll just have a bunch of hashes that we know are important [20:51:54] but thats fine if we don't intend to look into the features and do that analysis [20:51:54] Right. i agree with that. I was thinking about that a while ago and had some ideas. E.g. that a "hash" be a complex object that remembers it's underlying representation. [20:52:18] codezee, maybe not as a standard practice but as an analysis step. [20:52:46] halfak: then we're typically moving towards a dict based approach where we have built a vocab in a pre-processing step mapping words to indices [20:53:04] if i get the complex object stuff correctly [20:54:16] there seems to be tradeoff between hashing benefits and ability to get back the words... [20:55:00] i will also spend some time on this in literature to find if there's something of a middle ground [20:55:23] btw, looks like a good research discussion topic [20:58:12] halfak: Was I wrong that features= shows a few FeatureVectors? Looking at the code, now I think those are just lengths of a vector, therefore type Int. [20:59:15] right. Was trying to figure out what you were talking about awight ;) [20:59:22] lolol codezee ^ fyi [20:59:35] :P [20:59:46] omg I thought we were eliding vectors in the display and just showing the length as an informational thing. [20:59:59] We have FeatureVectors implemented and tested but then I needed to drop the kind of work that codezee is doing [21:00:01] I…. I’m going to deploy some things now. [21:00:06] \o/ [21:00:08] lol [21:00:27] awight, that's totally a thing that should happen if you were right [21:00:34] but it owuld include more info [21:01:04] haha that’s generous to cover for me [21:02:08] Gl on deploying awight... try not to break anything this time [21:02:41] Zppix: eh? I was very much planning to get gastrointestinal on it [21:03:10] awight: your not a doctor your a developer... oh what the hell do you want a scapel or a machete [21:06:09] Zppix: Stop calling me shirley. My drinking problem currently looks like: https://www.youtube.com/watch?v=pl4plPGRG8o [21:06:57] Legit me rn [21:22:25] Awight and srrodlund i was wondering if we could schedule a time on irc to sit and write https://etherpad.wikimedia.org/p/ORES_FAQ_Email together if possible? [21:25:57] Zppix: I’m not sure that’s a productive use of time any more. There’s a draft kicking around now and I think we’re just gonna send it. [21:26:30] Ok [21:32:34] wiki-ai/revscoring#1298 (selectors-fix - a451c4d : Sumit Asthana): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/304965637 [21:38:08] Deploying ORES. This could get turbulent, there’s been an issue with updating the server wheels. [21:38:16] s/wheels/python libs/ [21:48:08] Canary good, continuing. [21:48:53] Sounds like you need a mechanic for the "server wheels problem" [21:49:39] Zppix: Me fixing our server wheels, http://www.northernsoul.me.uk/wp/wp-content/uploads/2013/11/charlie-1.jpg [21:50:27] awight: watch out *crunch* [21:50:49] lol [21:51:25] *love* that movie [21:51:30] Lol [22:17:47] Can i turn off the nuclores alarm yet awight? Or did scap take a pass again? [22:18:09] Zppix: What’s this? [22:18:28] A nuclores alarm? [22:18:32] Thanks if you paused icinga2—the production cluster is done now. [22:18:45] It’s on to the ores* new clusters [22:19:33] Yeah puppet was acting up bc chase broke it so silenced it so it wouldnt be spammy [22:20:12] is icinga still seeing failures? [22:20:51] Icinga2? [22:20:59] You should have access [22:21:11] gerrit-icinga.wmflabs.org awight [22:21:22] If you need your creds reset letme know [22:29:03] awight, deploy went well? [22:29:07] * halfak was meeting [22:29:23] halfak: I’m too superstitious to say it, but without a hitch! [22:29:37] I smoke-tested T179712 and it works [22:29:37] T179712: ORES 500s when model_info lookup fails due to a key error - https://phabricator.wikimedia.org/T179712 [22:30:28] Verified T179711 works, too. [22:30:28] T179711: ORES 500 errors on a threshold lookup request - https://phabricator.wikimedia.org/T179711 [22:30:46] And the Revscoring 1 ghosts have been purged. [22:31:02] * awight rinses away ectoplasm [22:34:34] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3776131 (10Aklapper) @awight, @Ladsgroup: Please provide information / links how to create "qunit tests for wikilabels" and how to "integrate with travis". https://ww... [22:36:16] 8 days til GCI i cant wait [22:36:50] 10Scoring-platform-team, 10articlequality-modeling, 10Easy, 10Google-Code-in-2017, 10artificial-intelligence: Implement feature for detecting clumps of text that lack references - https://phabricator.wikimedia.org/T174384#3776136 (10Aklapper) @awight, @Ladsgroup: Please provide information / links to mor... [22:37:29] Zppix: whatchya doing? [22:37:45] Nothing important why? [22:38:39] Awight ^ [22:38:42] It sounded exciting for a minute [22:39:57] Awight im doing mentoring for docs, and easier tasks [22:40:09] (Its my 1st year of mentoring GCI) [22:40:21] I thought u needed me to do someyhing xD [22:40:26] Something [22:42:11] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: Error messages should not contain relative paths or error codes - https://phabricator.wikimedia.org/T175726#3776174 (10Aklapper) I assume a contributor is expected to provide a pull request to https://github.com/wiki-ai/wikilabels Import... [22:43:08] Zpppix: Thanks for taking that on! I wish I could find more time to mentor—not that I’m good at it, but it’s always fun to force myself to learn things in order to act teachy. [22:44:02] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776181 (10matmarex) Looks like a typo in ORES configuration? [22:44:15] Awight something broken [22:44:19] harrrr [22:44:32] Ruwp seems to be mainly affected atm [22:44:37] T181006 [22:44:38] T181006: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006 [22:44:58] halfak: Looks like we caught some exciting configuration via the new changes. ^ [22:45:11] awight: typos? [22:45:12] arg [22:45:18] This previously should have been throwing 500s, so I’m surprised it showed up just now. [22:45:47] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776141 (10Catrope) Looks like `22:27 awight@tin: Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring... [22:46:15] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776193 (10stjn) [22:46:36] awight: its anon and logged in users [22:47:20] Ill watch enwiki hopefully nothing happens there [22:47:27] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776141 (10awight) @Catrope Definitely caused by my deployment. The strange thing is, what we deployed was a fix to... [22:48:17] halfak: So now we know what Ext:ORES does when it sees a null :) [22:48:30] aha. Yeah. Was worried about that [22:48:44] amirite that we should push a config fix rather than fix Ext:ORE? [22:49:01] awight: need to teach it subtlety and manners still lol [22:49:14] We should see this in beta, right? [22:49:22] hi [22:49:24] awight, I dunno [22:49:25] this needs to be fixed ASAP [22:49:27] yeah but ruwiki might not be enabled there. [22:49:31] revert first please [22:49:36] ^ [22:49:37] legoktm: kk [22:49:39] then deploy it to mwdebug and test there [22:50:17] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776202 (10Krinkle) [22:50:25] Maybe its something ru-specific so thats why it didnt show in beta? [22:50:33] Zppix: that is correct [22:51:41] I think we have a flaw beta testing system, we ought to test on more then just an en-like config perhaps? Test it on multi configs [22:56:06] Anything i can do? [22:59:12] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776239 (10Krinkle) [23:02:56] Just got out of my last meeting. [23:03:03] awight, I'm on standby if you need anything [23:06:28] ty [23:09:33] * halfak submits codezee's recommendations. [23:10:43] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 2 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776287 (10MaxBioHazard) Still don't working. When it will be fixed? [23:15:56] awight: after this is resolved, I'd like to ask you to write down somewhere why ORES is (seemingly) causing spikes of 500s every other week or so. What decisions in how it was written are causing this and how can we address it (so that it fails less spectacularly)? Thanks. [23:26:27] greg-g: Will do, thanks for the nudge. [23:26:49] awight: and i guess at this point, just turn it into an incident report [23:27:31] 10Scoring-platform-team, 10Release-Engineering-Team: Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776331 (10awight) [23:27:59] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776344 (10greg) [23:28:00] https://ru.wikipedia.beta.wmflabs.org/wiki/Main_Page [23:28:45] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges don't work on ruwiki - https://phabricator.wikimedia.org/T181006#3776346 (10Krinkle) [23:28:48] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776331 (10Halfak) I'm not sure why this issue didn't show up with https://ru.wikipedia.beta.wmflabs.org/w... [23:29:42] It seems restricted to rc and wl halfak [23:29:46] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3776141 (10Krinkle) [23:29:58] Yeah. RC works theree [23:30:40] Maybe ruwiki in prod has some weird thing they added [23:30:53] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3776357 (10Krinkle) [23:33:09] Looks like this is a fallback failure in the new RecentChanges/Watchlist [23:33:23] ORES should not be able to take down Watchlist/RecentChanges. [23:33:38] Also, we have a SLOW deploy process because of the raw number of servers. [23:34:00] If we could parallelize, we could do this way faster and Ops wouldn't be yelling at awight for something that isn't his fault. [23:34:09] ^ agreed [23:34:34] I think its the stress from the train rollback 2 times now [23:37:36] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3776141 (10Halfak) I've confirmed trhat both Wikis have recovered. So, for clarity, it seems t... [23:38:34] Wait... RuntimeException: Unable to parse threshold: [..] at /srv/mediawiki/php-1.31.0-wmf.7/extensions/ORES/includes/Stats.php on line 277 why is it pulling from .7 after the train had it at .8? [23:40:09] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776368 (10awight) [23:41:03] halfak: ty but it totally was my fault. Writing about why that is, now :) [23:41:39] Still don't think it's your fault. Thanks for your work on this. [23:41:59] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776331 (10awight) [23:42:17] I'm going to step away now. Seems stable. I'll be around to help with docs tomorrow AM. [23:42:20] I’m going to get that year of my life back shortly, once I hit 100,000 points :p [23:42:21] awight, ^ [23:42:38] yep I’ll listen to chatter for another few minutes then I’m out, too. [23:42:54] cool. Have a good one. [23:42:55] o/ [23:43:24] *cough* not ur fault scap is just a dick [23:43:28] Awight ^ [23:43:58] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776380 (10awight)