[00:12:28] <wikibugs>	 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10artificial-intelligence: Review training set to check strange examples of labels - https://phabricator.wikimedia.org/T171497#3468626 (10Natalia) Here are some results of reviewing labels Damaging/Goodfaith and Damaging/Badfaith for enwiki and ruwik...
[07:08:28] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017), 10Patch-For-Review: Summarize what it will take to separate UI and infrastructure for ORES Extension - https://phabricator.wikimedia.org/T167908#3469009 (10awight)
[07:14:57] <wikibugs>	 (03CR) 10Awight: [C: 04-1] "I want to merge this as is, but the requirements have changed slightly (sorry!).  See my change to the bug description...  Now I'm thinkin" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/359364 (https://phabricator.wikimedia.org/T167908) (owner: 10Catrope)
[08:16:45] <wikibugs>	 (03CR) 10Ladsgroup: [C: 031] "It looks okay to me with valid use cases but I'm a little bit worried about the storage problems it might cause, in prod we use InnoDB as " [extensions/ORES] - 10https://gerrit.wikimedia.org/r/367449 (owner: 10Catrope)
[13:37:25] <icinga-wm>	 PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert.
[13:40:33] * halfak looks
[13:41:52] <halfak>	 Looks like we're fine. 
[13:42:14] <halfak>	 But holy moley! there was a bit batch job that hit us yesterday.
[13:52:35] <icinga-wm>	 RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting.
[14:18:28] <wikibugs>	 10Scoring-platform-team, 10ORES: Set up larger ores-compute instance - https://phabricator.wikimedia.org/T169809#3470179 (10Andrew)
[14:18:33] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Cloud-VPS (Quota-requests), 10User-bd808: Request increase quota for ores-staging to 52GB RAM - https://phabricator.wikimedia.org/T169811#3470177 (10Andrew) 05Open>03Resolved a:03Andrew
[14:42:20] <halfak>	 OMG we planned a deployment for today. 
[14:42:24] <halfak>	 I don't see any pull requests. 
[14:42:27] <halfak>	 I'm going to go check on that. 
[15:59:39] <halfak>	 o/ Amir1 
[15:59:45] <halfak>	 still want to do that deployment today?
[15:59:59] <Amir1>	 Since we announced it, I think so
[16:00:48] <halfak>	 Any PR's or patchsets?
[16:00:58] <halfak>	 If not, I'll get that out of the way ASAP
[16:06:26] <Amir1>	 halfak: I couldn't get it out of the door because I had to clone lots of things into misc and took forever 
[16:08:57] <halfak>	 Amir1, gotcha.  I'll work on it. 
[16:13:16] <halfak>	 Amir1, can you double check https://ores-staging.wmflabs.org/
[16:14:02] <awight-shoddy>	 Is that a production or labs alert?
[16:14:08] <halfak>	 ?
[16:14:24] <halfak>	 awight-shoddy, which alert
[16:14:42] <awight-shoddy>	 Grafana 5xx
[16:15:02] <awight-shoddy>	 I'm on a phone and busy deleting forensics.  It recovered already?
[16:15:31] <halfak>	 Oh yeah.  That fired and recovered a few hours ago. :) 
[16:15:58] <halfak>	 I was thinking that I want to know what is happening.  We shouldn't have ChangeProp getting 500s at all. 
[16:17:18] <awight-shoddy_>	 Nvm, this was a blip hours ago
[16:17:36] <awight-shoddy_>	 Good to know grafana layout work ok 
[16:17:48] <awight-shoddy_>	 Works on the old phone
[16:17:53] <halfak>	 :D  
[16:21:33] <Amir1>	 halfak: my laptop died, I'm back now
[16:21:34] <Amir1>	 sorry
[16:21:44] <halfak>	 Hey!  Looks like the staging deploy is OK
[16:21:51] <halfak>	 I'm working on patchset for prod. 
[16:24:14] <awight-shoddy_>	 I didn't even know there was a deployment today.  Fwiw I'll be situated in 35 minutes
[16:24:46] * awight-shoddy_ revels in continuity between nick, actual cell service
[16:25:00] <awight-shoddy_>	 , and care of duty
[16:25:14] <Amir1>	 halfak: this needs mediawiki deployment too
[16:25:27] <Amir1>	 Remind me to get it through swat too
[16:25:28] <halfak>	 Amir1, oh?  Why is that? 
[16:25:30] <halfak>	 Amir1,  https://gerrit.wikimedia.org/r/367695
[16:25:37] <Amir1>	 the thresholds have changed 
[16:25:50] <halfak>	 awight-shoddy_, it's on your calendar :P 
[16:25:58] <Amir1>	 not much but out still needs some adjustments for ores review tool
[16:26:17] <halfak>	 Amir1, oh yeah!  Gotcha. 
[16:26:40] * awight-shoddy_ squints between fingers
[16:39:32] <fajne>	 halfak: so, I changed False to True, then rebuilt and retrained the whole thing (make enwiki_models, make enwiki_training_reports) but enwiki_goodfaith.md failed..
[16:40:14] <halfak>	 fajne, great progress.  Let me know what that error is.  I'll need to be AFK soon but I'll look at it while I eat lunch
[16:46:26] <halfak>	 OK  Beta deploy time. 
[16:46:38] <halfak>	 Amir1, do you have the facilities to check on our beta deploy not 'sploding anything?
[16:47:10] <Amir1>	 nope, except the outside endpoint (ores-beta.wmflabs.org)
[16:47:17] <Amir1>	 I need to go
[16:47:19] <Amir1>	 be back in a sec
[16:47:31] <fajne>	 halfak: Makefile:463: recipe for target 'tuning_reports/enwiki.goodfaith.md' failed make: *** [tuning_reports/enwiki.goodfaith.md] Error 1 make: *** Deleting file 'tuning_reports/enwiki.goodfaith.md'
[16:54:01] <halfak>	 fajne, that's just makefile output.  I need the error that caused it. 
[16:56:10] <halfak>	 Amir1, ores-beta deploy is complete. 
[16:56:36] <halfak>	 Looking at https://grafana.wikimedia.org/dashboard/db/ores-extension?orgId=1
[16:56:47] <halfak>	 Nothing looks bad. 
[16:56:58] <halfak>	 Woops.  https://grafana-labs.wikimedia.org/dashboard/db/ores-extension?orgId=1
[16:57:04] <halfak>	 Also nothing bad.  Generally nothing though
[17:00:49] <fajne>	 halfak: yep.. looking for it
[17:01:03] <awight>	 halfak: Can I do deployey things
[17:02:07] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10demon)
[17:04:19] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470987 (10demon)
[17:20:13] <fajne>	 halfak: what was the service that let you generate a link for a text you copy paste in?
[17:20:26] <halfak>	 pastebin.ca
[17:20:30] <halfak>	 ^ my fav
[17:22:39] <fajne>	 hm.. another one?
[17:22:53] <fajne>	 (this gives me an error and sorry)
[17:23:51] <fajne>	 halfak: oh, it finally worked https://pastebin.ca/3846508
[17:24:43] <halfak>	 fajne, that's a warning. 
[17:24:44] <halfak>	 It's OK
[17:25:13] <fajne>	 that's a ValueError
[17:25:34] <halfak>	 Right
[17:25:45] <fajne>	 also, could it be that a test set contains the same kind of bug as the training set did?
[17:25:55] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10bd808) #striker could use this too. It has the same sort of wheel blob repo as ORES.
[17:27:53] <fajne>	 halfak: I got 6 exceptions like this and no more errors
[17:28:23] <halfak>	 Those don't cause the process to fail
[17:28:25] <halfak>	 they are expected. 
[17:28:36] <fajne>	 it buit the tunung report for damaging model and reverted one but failed for goodfaith
[17:28:46] <halfak>	 Right.  What error cause it to fail?
[17:28:50] <wikibugs>	 10Scoring-platform-team, 10Release-Engineering-Team, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Split editquality repo to two repos, one with full history, one shallow - https://phabricator.wikimedia.org/T170967#3471090 (10awight)
[17:28:51] <halfak>	 It should be the last one listed. 
[17:29:08] <fajne>	 i gave you the very last line in the biginning
[17:29:29] <awight>	 When you’re all less busy… Anyone know what caused the 13:30-13:40 glitch?
[17:30:44] <fajne>	 halfak, https://pastebin.ca/3846518
[17:31:33] <halfak>	 Looks like tune is not parameterized right
[17:31:49] <halfak>	 It's missing a line 
[17:31:53] <halfak>	 For <label>
[17:32:04] <halfak>	 Accidental deletion?
[17:33:31] <fajne>	 what if i just download the repo again/
[17:34:19] <halfak>	 fajne, just add the line back :) 
[17:34:36] <halfak>	 In your makefile for that "tune" rule. 
[17:34:50] <halfak>	 It's also possible that it was broken in the repo :/
[17:36:45] <icinga-wm>	 PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert.
[17:36:55] <halfak>	 ^ Likely due to deploy
[17:37:05] <wikibugs>	 10Scoring-platform-team, 10editquality-modeling, 10Release-Engineering-Team (Watching / External), 10User-Ladsgroup, 10artificial-intelligence: Split editquality repo to two repos, one with full history, one shallow - https://phabricator.wikimedia.org/T170967#3471163 (10greg) See also: {T171619}
[17:40:02] <halfak>	 Confirmed. 
[17:40:04] <halfak>	 Deploy realted
[17:40:08] <halfak>	 *related
[17:41:12] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10awight) I don't want to hijack this task, but for the record our binary problem is much more severe in the `editquality` repo, wh...
[17:43:09] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10greg) Yeah, let's do it for all the repos with binaries :) And we'll figure out something re labs + prod re git-fat access.
[17:44:12] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES, 10Release-Engineering-Team, 10Scap: ORES should use git-fat for wheel deployments - https://phabricator.wikimedia.org/T171619#3470967 (10Halfak) Where are the best docs for setting this up.  Will it work with out github repos?  We do development mainly against them...
[17:44:27] <awight>	 halfak: weirdly, that grafana alert doesn’t correspond to anything in the graph.  yet?  https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1500994470546&to=1501002873413&panelId=23&fullscreen
[17:44:30] <awight>	 ooooh my time axis is zoomed
[17:44:30] <awight>	 D’OH
[17:44:37] <halfak>	 :D
[17:44:49] <halfak>	 OK I'm declaring victory and running away for lunch
[17:44:52] <halfak>	 Back in ~ an hour
[17:45:56] <awight>	 enjoy the well-earned march up and down the square
[17:50:45] <icinga-wm>	 RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting.
[17:59:23] <awight>	 RoanKattouw: I might take a crack at https://gerrit.wikimedia.org/r/#/c/359364/1, assuming you’re not looking at it by coincidence?
[17:59:50] <RoanKattouw>	 awight: Go for it
[18:01:52] <awight>	 :D  Can I can your opinion of the new plan’s sanity, btw?  I want the boolean to disable the entire UI, both legacy and RC filters ORES
[18:02:02] <awight>	 s/can/get/
[18:21:33] <wikibugs>	 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-Recent-changes, and 2 others: Read timeout on Special:RecentChanges with complex query - https://phabricator.wikimedia.org/T168096#3356056 (10Catrope) Clarification: this query is looking for edits tagged...
[18:26:03] <awight>	 RoanKattouw: quick question above ^
[18:46:59] <RoanKattouw>	 awight: Yeah sounds good to me
[18:47:03] <RoanKattouw>	 Sorry for spacing
[18:47:22] <RoanKattouw>	 It probably isn't too bad an idea to do that in routine deployment anyway
[18:47:29] <RoanKattouw>	 Yesterday I deployed the extension to sqwiki and rowiki
[18:47:52] <RoanKattouw>	 I just put the config in and enabled it, then was surprised when RC errored out, and went "oh right I have to create the DB tables don't I"
[18:47:54] <awight>	 RoanKattouw: great, thanks for the “sanity” !
[18:47:57] <RoanKattouw>	 (and then populate them with scores too)
[18:48:17] <awight>	 lol yes exactly, I love when the code and feature are loosely coupled
[18:48:32] <RoanKattouw>	 Thankfully this was all on mwdebug1002 so people weren't being offered a broken beta feature, but I did realize that it's important to populate the tables a bit before actually putting the UI in front of users
[19:23:43] <halfak>	 \o/  I'm excited for this de-coupling
[19:23:52] <halfak>	 fajne, any luck with that tuning report?
[19:28:13] <awight>	 urrrgh.  3 hours later, I can finally develop on ext-ORES
[19:28:35] <awight>	 legoktm: btw PROPS for the mediawiki.deb, and thank you
[19:29:11] <awight>	 I just used it to get a mw-ext dev environment in a virtualbox and quite happy not having to do the arcane apache etc. steps
[19:56:00] * halfak finally starts work on the ORES stress test
[20:06:41] <wikibugs>	 (03PS2) 10Awight: Allow filter UI to be turned off [extensions/ORES] - 10https://gerrit.wikimedia.org/r/359364 (https://phabricator.wikimedia.org/T167908) (owner: 10Catrope)
[20:06:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Allow filter UI to be turned off [extensions/ORES] - 10https://gerrit.wikimedia.org/r/359364 (https://phabricator.wikimedia.org/T167908) (owner: 10Catrope)
[20:18:44] <wikibugs>	 (03PS3) 10Awight: Allow filter UI to be turned off [extensions/ORES] - 10https://gerrit.wikimedia.org/r/359364 (https://phabricator.wikimedia.org/T167908) (owner: 10Catrope)
[20:19:32] <fajne>	 hal
[20:20:24] <awight>	 I hadn’t noticed until now that halfak might be related to https://en.wikipedia.org/wiki/HAL_9000
[20:20:24] <fajne>	 halfak: sorry, yep, found that line, also found one forgotten edit to max_features, retrainig now
[20:20:34] <halfak>	 sssh
[20:20:44] <halfak>	 \o/ :) 
[20:20:48] <halfak>	 fajne, ^ 
[20:25:59] <wikibugs>	 (03CR) 10Awight: "PS 3: Toggle the entire UI with this new config.  Rebase." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/359364 (https://phabricator.wikimedia.org/T167908) (owner: 10Catrope)
[20:45:37] <awight>	 whew!  the fi.flaggedrevs model is training
[20:45:52] <halfak>	 \o/
[20:47:23] <awight>	 I’d really like to get an email at the end of the job...
[20:50:25] <fajne>	 halfak: 0.922 for damaging and 0.928 for goodfaith :(
[20:50:39] <halfak>	 roc_auc?
[20:51:01] <fajne>	 aha
[20:51:30] <halfak>	 That's a substantial improvement for goodfaith
[20:51:36] <halfak>	 I did not expect damaging to change at all :) 
[20:51:37] <fajne>	 seriously?
[20:51:41] <halfak>	 yup
[20:51:49] <halfak>	 Current is 0.909
[20:52:05] <fajne>	 \o/
[20:52:09] <halfak>	 :DDD
[20:53:32] <awight>	 holy cow.  This is just ruwiki += sentiment features?
[20:54:33] <halfak>	 awight, I think fajne is working on training set cleanup
[20:54:52] <awight>	 dang!
[20:59:19] <fajne>	 awight: Aaron, inspired by false positives I brought from the training set, found a bug in the fetch_labels.py. That's all. 
[20:59:35] <awight>	 I wonder how much of the gain was that the model got better, vs. the testing labels becoming correct.  We could check that by testing against clean and unclean data...
[21:00:15] <awight>	 fajne: that’s the most awesome way to make progress, finding the leaks in the hull of the boat!
[21:04:46] <awight>	 fajne: Mind if I assign T171493 to you?
[21:04:46] <stashbot>	 T171493: Change "yes/no" in damaging_goodfaith form to "damaging/good" and  "good-faith/bad-faith" - https://phabricator.wikimedia.org/T171493
[21:04:52] <awight>	 or is that something else...
[21:04:56] <fajne>	 awight: the bug was that all unlabeled edits were considered badfaith. we changed to goodfaith. so, its training labels becoming closer to reality and consequently the model becoming correct
[21:05:27] <fajne>	 its the UI
[21:05:38] <awight>	 kk I think I follow
[21:05:42] <fajne>	 ^
[21:06:07] <fajne>	 so, no, i won't handle it)
[21:06:16] <awight>	 oic T171497
[21:06:16] <stashbot>	 T171497: Review training set to check strange examples of labels - https://phabricator.wikimedia.org/T171497
[21:06:30] <fajne>	 i did it
[21:06:40] <fajne>	 i believe, there is a report there
[21:06:46] <awight>	 I’m just trying to find the next thing to do, myself.  Don’t mind me :-)
[21:06:47] <fajne>	 a link to one
[21:06:55] <fajne>	 ))
[21:08:05] <awight>	 halfak: I’m looking for another task to pick up, but nothing in Review or Active is jumping out at me.
[21:08:13] <halfak>	 fajne, I'm a fan of you taking on that yes/no --> damaging/good bug
[21:09:52] <halfak>	 phab down?
[21:10:12] <awight>	 halfak: just you :)
[21:10:27] <halfak>	 Weird...
[21:10:55] <halfak>	 can't reach https://en.wikipedia.org/ either
[21:11:01] <halfak>	 but I can reach https://google.com
[21:14:19] <fajne>	 awight: can i drop by the office tomorrow morning?
[21:17:18] <wikibugs>	 10Scoring-platform-team, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Investigate small loss in fitness with the new data in fawiki - https://phabricator.wikimedia.org/T171386#3463133 (10Halfak) a:05Halfak>03Ladsgroup
[21:18:00] <halfak>	 awight, could you work on the meta ORES schema?  I think a strawman proposal would be a great next step. 
[21:18:08] <awight>	 lol http://downforeveryoneorjustme.com
[21:18:13] <awight>	 aha thanks, that sounds fun
[21:18:49] <halfak>	 I'll come in with some thoughts regarding what I think we have learned from fajne's analysis
[21:20:33] <wikibugs>	 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10ORES, 10Wikidata, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): ORES: Don't highlight changes propagated from Wikidata - https://phabricator.wikimedia.org/T168487#3472593 (10jmatazzoni)
[21:25:49] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Patch-For-Review, 10User-Ladsgroup: Move from mediawiki/services/ores/deploy to research/ores/deploy or research/ores/deploy-prod - https://phabricator.wikimedia.org/T139008#2416663 (10awight) @Halfak @Ladsgroup ping, this is for us.  Is it true that this task is just a n...
[21:26:45] <wikibugs>	 10Scoring-platform-team, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Flagged revs approve model to fiwiki - https://phabricator.wikimedia.org/T166235#3472639 (10awight)
[21:26:47] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-FlaggedRevs, 10ORES: Decrease FlaggedRevs backlog by using ORES predictions models - https://phabricator.wikimedia.org/T165848#3472638 (10awight)
[21:29:15] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Patch-For-Review, 10User-Ladsgroup: Move from mediawiki/services/ores/deploy to research/ores/deploy or research/ores/deploy-prod - https://phabricator.wikimedia.org/T139008#3472649 (10Halfak) 05Open>03Invalid Hmm... This task doesn't have the meaning it once had.  Le...
[21:29:51] <wikibugs>	 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10ORES, 10Collaboration-Team-Triage (Collab-Team-Q4-Apr-Jun-2017), 10MW-1.29-release-notes: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3100184 (10awight) See also T167910 and T167908
[21:32:34] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Epic: [Epic] Structured deployment of ORES - https://phabricator.wikimedia.org/T130369#2134015 (10awight) @Ladsgroup I see that all the subtasks are complete--should we revolve the epic?
[21:46:36] <awight>	 halfak: Can you paste me the meta-ores etherpads from your history?  This laptop doesn’t have ‘em.
[21:46:57] <awight>	 I’ll link from the tasks, once I have the addresses
[21:47:02] <halfak>	 https://etherpad.wikimedia.org/p/meta_ores_schema
[21:47:31] <awight>	 that’s the one.  thanks
[21:47:41] <paladox>	 I just found out that gerrit supports moving changes through rest api heh.
[21:47:53] <paladox>	 rest api is what all of gerrit's ui uses 
[21:48:30] <wikibugs>	 10Scoring-platform-team-Backlog, 10ORES: Design Meta ORES data storage schema - https://phabricator.wikimedia.org/T153152#3472735 (10awight)
[21:49:09] <awight>	 paladox: Yipes!  We could really shuffle the deck using the gerrit API, I’m sure…
[21:49:15] <paladox>	 lol
[21:49:27] <paladox>	 awight i am trying to add it to the ui in polygerrit
[21:49:42] <paladox>	 there's no way i can add the support for it in gwtui have no idea how to do it there
[21:49:45] <paladox>	 java is hard
[21:50:02] <awight>	 nice!  That’s a brave dogfeeding project.
[21:50:10] <paladox>	 awight also drafts are being removed
[21:50:15] <paladox>	 replaced with private
[21:50:21] <paladox>	 actually called that
[21:50:33] <paladox>	 and meant to do what drafts was supposed to do and be hidden
[21:50:50] <paladox>	 it also gains wip support
[21:51:00] <paladox>	 and a mute and ignore button too
[21:51:01] <awight>	 hrm.  I remember drafts being pretty counterintuitive
[21:51:20] <paladox>	 yeh, being replaced with privates in gerrit 2.15 :)
[21:51:25] <awight>	 Not sure I trust “improvements” either, though
[21:51:37] <paladox>	 i found it's an improvement
[21:51:48] <paladox>	 you can now set your change as a private or public or private again
[21:54:21] <awight>	 I guess I’ve been lacking a way to prevent people from reviewing WIP patches, so rely heavily on [tag] stuff in the commit title
[21:55:25] <harej>	 When a query is made to ORES about a past revision of a wiki page, is that query result cached? Is this a permanent or temporary cache?
[21:55:39] <halfak>	 temporary
[21:55:48] <halfak>	 Do you know what an LRU is?
[21:56:10] <harej>	 Hmm. So if I want to store revision quality information permanently, I should create my own data store?
[21:56:24] <paladox>	 awight with wip it will now remove it from reviewers dashboards
[21:56:29] <harej>	 I want to do longitudinal article quality research.
[21:56:33] <paladox>	 until you click the start review button
[21:56:40] <awight>	 harej: https://github.com/wikimedia/mediawiki-extensions-ORES/blob/master/includes/Cache.php
[21:56:50] <awight>	 halfak: Do we actually have a LRU cache?
[21:56:56] <halfak>	 harej, no reason you can't do longitudinal research
[21:57:02] <halfak>	 awight, yes.  Approximately. 
[21:57:15] <halfak>	 https://redis.io/topics/lru-cache
[21:57:29] <awight>	 ooh right there are two caches < harej 
[21:57:57] <halfak>	 harej, you can always get an old revision scored
[21:57:58] <awight>	 The ORES API is cached in Redis, but the MediaWiki Extension:ORES caches those results in MySQL
[21:58:08] <halfak>	 awight, +1
[21:58:20] <awight>	 harej must be talking about the API
[21:58:23] <halfak>	 Extension:ORES will limit the cache to revisions that appear in recentchanges
[21:58:26] <harej>	 halfak: I guess a better way of phrasing my question is, is it worth it for me to cache the data on my end so that I don't have to keep pinging ORES about the quality of revision 12345, or should it be fine for me to just retrieve the data from ORES every time?
[21:58:36] <harej>	 and yes I mean the original API
[21:58:38] <halfak>	 Just retrieve it every time :) 
[21:58:46] <harej>	 Damn really? Cool
[21:58:46] <awight>	 it makes us look good :p
[21:58:57] <halfak>	 until that proves intractable -- which I suspect it will not. 
[21:58:59] <awight>	 budget ~= API queries
[21:59:12] * awight is kicked by halfak 
[21:59:17] <harej>	 We like ORES. I published a research paper using ORES. So I am making ORES article quality reports a part of my new dashboard.
[21:59:32] <halfak>	 You published a research paper using ORES?  Cool!
[21:59:50] <harej>	 Well, a poster, which I think over time has been transmuted into a paper. I keep getting credited for things.
[21:59:59] <awight>	 harej: that’s awesome news!  link here :-)
[22:00:10] <halfak>	 \o/
[22:00:21] <harej>	 HaeB had a link and sassed me for not having it open access, even though it was completely outside my control
[22:00:42] <awight>	 haarrrgh
[22:00:46] * halfak pings ores.wmflabs.org for 600 requests per minute
[22:00:49] <halfak>	 Mwahahaha!
[22:00:59] <harej>	 wait is it ores.wmflabs or ores.wikimedia
[22:01:01] <awight>	 halfak: Sounds stressful!
[22:01:05] <halfak>	 wmflabs
[22:01:09] <awight>	 harewat
[22:01:11] <harej>	 I thought we productionized it
[22:01:13] <awight>	 wat and wat
[22:01:16] <halfak>	 Testing my stresstester in a safe place
[22:01:25] <awight>	 harej: you should query ores.wmo
[22:01:32] <halfak>	 +1
[22:01:47] <awight>	 irc async error :)
[22:02:16] <halfak>	 OK now to test the big'ne 
[22:02:20] <halfak>	 I might take ORES down. 
[22:02:23] <halfak>	 Hold onto your pants. 
[22:02:40] <awight>	 what’s happening now?
[22:03:09] <halfak>	 pants holding
[22:03:11] <halfak>	 :)  
[22:03:18] <halfak>	 I'm running my stress tester against prod
[22:03:22] <halfak>	 DOS tool
[22:03:23] <awight>	 oh dear
[22:03:48] <awight>	 https://en.wikipedia.org/wiki/Low_Orbit_Ion_Cannon
[22:05:06] <halfak>	 It's alive!
[22:05:23] <halfak>	 And it's doing a pretty good job of hitting ORES at 600 req/sec
[22:05:30] <halfak>	 *10 reqs/sec
[22:05:42] <halfak>	 600 req/min
[22:05:53] <awight>	 So you’re roughly doubling volume?
[22:05:57] <halfak>	 yup
[22:06:03] <awight>	 verra nice
[22:06:08] <halfak>	 Just to codfw though
[22:06:32] <halfak>	 0 errors :) 1887 revisions scored
[22:06:54] <awight>	 Seems to be hitting eqiad as well.
[22:07:17] <awight>	 oh strange—or our graph is wrong?
[22:07:28] <awight>	 https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1501019480504&to=1501020393998
[22:08:23] <halfak>	 Interesting.  Thought my geography would have CODFW or bust
[22:08:34] <awight>	 I think you’re hitting codfw
[22:08:42] <awight>	 but the eqiad line is incorrectly tallied.
[22:08:54] <awight>	 select scb1000*, you’ll see no extra traffic
[22:09:21] <halfak>	 Aha!
[22:09:56] <halfak>	 Refresh
[22:10:10] <awight>	 meanwhile, I can’t do any logging in
[22:10:26] <awight>	 Did you fix it?
[22:10:32] <halfak>	 yup
[22:10:43] <awight>	 :D
[22:11:03] <awight>	 that was useful already
[22:11:22] <awight>	 How are you choosing revision IDs?
[22:11:32] <awight>	 I’m surprised that the cache hit rate is unchanged.
[22:11:32] <halfak>	 right now, just a little bit of our training set
[22:11:47] <halfak>	 awight, I purposefully circumvented cache
[22:11:50] <halfak>	 So it doesn't count :) 
[22:11:59] <halfak>	 "?features" skips the cache 
[22:12:07] <wikibugs>	 10Scoring-platform-team, 10ORES: Design Meta ORES data storage schema - https://phabricator.wikimedia.org/T153152#3472853 (10awight)
[22:12:10] <awight>	  ./o\
[22:12:23] <awight>	 btw ^ that is our DoS vector :p
[22:12:33] <halfak>	 yup :S
[22:12:44] <halfak>	 ragesoss, is going to DOS us with that soon too
[22:13:04] <halfak>	 Oh wait... I'm not sure they need to.  We still cache feature injections.
[22:13:05] <ragesoss>	 no promises
[22:13:08] <halfak>	 :) 
[22:13:12] <ragesoss>	 but I'll do my best.
[22:13:15] <halfak>	 haha!
[22:14:55] <ragesoss>	 I'm not actually DOSing ores, am I?
[22:15:13] <awight>	 halfak: Doesn’t look like you’re even moving the needle on those boxes: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Service%2520Cluster%2520B%2520codfw&tab=m&vn=&hide-hf=false
[22:15:19] <halfak>	 No.  :)  Just any time you use the "?features" feature. 
[22:15:30] <halfak>	 ragesoss, ^ 
[22:15:38] <ragesoss>	 yeah, all my requests, just about, do use ?features.
[22:15:58] <halfak>	 awight, we do a lot of IO blocking
[22:16:04] <halfak>	 That must be why
[22:16:17] <halfak>	 ragesoss, if it's you typing the URL, no big deal. 
[22:16:23] <halfak>	 If you're scoring 900k edits, bad times
[22:18:27] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Patch-For-Review: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3472885 (10Halfak)
[22:19:11] <ragesoss>	 halfak: probably a few thousand per day.
[22:19:52] <awight>	 halfak: Why would you be causing disk io?
[22:19:58] <awight>	 The models will all be cached in memory...
[22:20:06] <halfak>	 awight, not disk.  network
[22:20:14] <halfak>	 ragesoss, 1k per day is nothing :) 
[22:20:22] <halfak>	 10k per day would be fine too
[22:20:45] <awight>	 aah.  yeah network roughly doubled, to 8MB/s
[22:20:55] <awight>	 turn it to 10x and see what happens…
[22:21:04] <paladox>	 lol
[22:21:05] * awight is a fan of destructive testing
[22:21:10] <paladox>	 lol
[22:21:27] <halfak>	 nope nope nope ;P
[22:21:34] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Patch-For-Review: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3472889 (10Halfak) https://github.com/wiki-ai/ores/pull/221  I've confirmed that this works pretty well with a controlled test against prod.  See the bump in codfw traffic h...
[22:21:55] <ragesoss>	 that's a guess. between both dashboards, I would guess we're under 10k / day, but would not be surprised if it was higher... but it may be considerably higher in the fall, since we now have some features that (as currently implemented) can potentially make several ores calls each time a user loads up a course page on the dashboard.
[22:22:22] <halfak>	 ragesoss, make sure you keep a descriptive user-agent with requests and we'll deal with it when it is a problem. 
[22:22:30] <ragesoss>	 we'll probably adjust that to only load after the user clicks on the specific assignment.
[22:22:43] <halfak>	 ragesoss, I'm sure that'll do wonders :) 
[22:22:44] <ragesoss>	 okay. I'll double check the user agent.
[22:23:00] <ragesoss>	 right now, load from that feature is very low anyway.
[22:23:42] <ragesoss>	 summer courses are few, dashboard traffic is light, and that feature is en.wiki-only for P&E Dashboard.
[22:26:16] <awight>	 halfak: For comparison, the busiest MW DB box is db1079, and it’s going at about 8MB/s, summing in+out.  Our busiest box during the load test was at 3MB/s total net traffic.
[22:26:53] <awight>	 Remind me where we serve Redis?
[22:27:09] <halfak>	 oresrdb*
[22:27:40] <halfak>	 ragesoss, ack.  If you are ever worried about your load, swing by here and let us know and we'll take a look
[22:27:54] <halfak>	 FWIW, you're one of our cherished users, so we'll figure out something good :) 
[22:28:03] <ragesoss>	 *I'm* not worried.
[22:28:04] <ragesoss>	 :)
[22:28:22] <halfak>	 Hmm... looks like my stress tester isn't tracking granular errors quite right. 
[22:29:25] <awight>	 oh right.  There’s nothing happening on oresrdb cos we were bypassing cache...
[22:29:56] <awight>	 I spoke too soon.  https://ganglia.wikimedia.org/latest/graph_all_periods.php?h=oresrdb2001.codfw.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2&st=1501021770&g=network_report&z=large&c=Miscellaneous%20codfw
[22:30:36] <halfak>	 We use those machines to run the celery queue too
[22:30:46] <halfak>	 The celery queue was *very active* during this time :) 
[22:30:49] <awight>	 aha, ty
[22:31:35] <awight>	 This might be the hot spot in our system, it’s a much more exciting jump than any of the other metrics
[22:32:29] <halfak>	 Woo!  Caught my error issue
[22:33:38] <halfak>	 Woops.  Left a debugging print in there
[22:41:11] <awight>	 Currently testing the fiwiki.flaggedrevs model….
[22:41:25] <awight>	 I’m gonna commute and poke at the meta-ores schema later.
[22:41:30] <awight>	 \o
[22:55:52] <halfak>	 o/
[23:52:31] <fajne>	 halfak: if you have a spare minute, what may cause this error https://www.pastebin.ca/3846609  ?