[00:07:09] <awight>	 taking a walk...
[01:46:42] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119091 (10awight) I ran a test on deployment-ores01.deployment-prep.eqiad...
[01:48:00] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119092 (10awight)
[02:02:17] <wikibugs>	 10Scoring-platform-team, 10ORES: Spike in ORES errors every 24hr - https://phabricator.wikimedia.org/T191864#4119099 (10awight)
[02:48:38] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119135 (10mmodell) >>! In T181071#4118685, @awight wrote: > I can't tell...
[02:52:42] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119136 (10mmodell)  >>! In T181071#4119091, @awight wrote: > There's no ....
[04:38:12] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4119223 (10awight) The weird part about this is just that we've been deplo...
[08:56:32] <wikibugs>	 10Scoring-platform-team (Current), 10Beta-Cluster-Infrastructure, 10Recommendation-API, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): What to do with deployment-sca03? - https://phabricator.wikimedia.org/T184501#4119535 (10EddieGP) 05Open>03Resolved a:03EddieGP That instance...
[10:24:36] <wikibugs>	 10Scoring-platform-team, 10ORES, 10articlequality-modeling, 10artificial-intelligence: Rename "wikiclass" to "articlequality" - https://phabricator.wikimedia.org/T181820#4119763 (10Ladsgroup) Yes. I just wanted to do the easiest thing possible :D
[13:44:33] <halfak>	 o/
[13:44:42] * halfak hacks on the bot model paper
[13:58:09] <halfak>	 Arg just realized I have a review due today. 
[14:47:59] <halfak>	 Man.  I
[14:48:03] <halfak>	 'm a hard reviewer. 
[15:10:36] <wikibugs>	 10Scoring-platform-team, 10ORES, 10articlequality-modeling, 10artificial-intelligence: Rename "wikiclass" to "articlequality" - https://phabricator.wikimedia.org/T181820#4120495 (10Halfak) OK, I'll wait for you to make a PR then.  I'll ping before I start work if I beat you to it.
[16:22:37] <tgr>	 awight: I get "ImportError: No module named 'editquality'
[16:22:51] <tgr>	 " when I try to build models after a rebase
[16:22:59] <tgr>	 any idea where to look?
[16:23:15] <awight>	 yes I think that’s just pip install -e . ?
[16:23:16] <tgr>	 I'm in the venv, I have run pip install recently
[16:23:44] <tgr>	 not -e though
[16:26:27] <tgr>	 same error after install -e .
[16:26:39] <awight>	 that’s surprising
[16:26:40] <awight>	 hum
[16:27:01] <tgr>	 no, wait, this is a different error actually
[16:27:22] <tgr>	 pkg_resources.ContextualVersionConflict: (scipy 0.17.1 (/srv/home/tgr/ores/venv/lib/python3.5/site-packages), Requirement.parse('scipy>=0.18.1'), {'gensim'})
[16:27:48] <awight>	 rats, lemme take a look
[16:28:09] <awight>	 try pip install -r requirements.txt
[16:28:25] <awight>	 I suppose you did that already
[16:28:30] <tgr>	 requirements.txt for revscoring says scipy >= 0.13.3, < 1.0.999
[16:28:42] <tgr>	 editquality doesn't mention it at all
[16:29:07] <awight>	 yes, I’m starting to think that the issue is that we’re too lenient about the revscoring requirement
[16:29:28] <awight>	 one thing you could do is, clone the revscoring repo, grab the head, and pip install -e that into the same virtualenv.
[16:29:51] <tgr>	 something is requiring >=0.18.1 though, where does that come from?
[16:30:03] <awight>	 That would be the gensim package, I believe
[16:30:38] <awight>	 It’s a new requirement and I’m surprised it’s in your dependency tree, I thought it was only included in the drafttopic repo.
[16:31:14] <tgr>	 it's in revscoring apparently
[16:31:52] <awight>	 wat
[16:32:12] <tgr>	 I suppose Python doesn't do recursive dependency resolution?
[16:32:13] <awight>	 I just pulled the revscoring head and didn’t see it
[16:32:14] <tgr>	 a bit lame
[16:32:17] <awight>	 I believe it does
[16:32:29] <awight>	 I think what you have is a conflict ;-)
[16:32:57] <awight>	 my revscoring head is f60e86ac89837b8fe4a13a2a3e400b4ff39b8c56, Mar 16
[16:33:08] <awight>	 oooh
[16:33:13] <awight>	 requirements.txt isn’t alphabetical
[16:33:16] <awight>	 I see gensim now.
[16:33:55] <tgr>	 not a real conflict though, gensim wants scipy>=0.18.1, revscoring wants scipy >= 0.13.3
[16:34:07] <tgr>	 pip just did not bother to update it
[16:35:09] <awight>	 strange business!
[16:36:23] <awight>	 This is silly, but you might be happy installing the wheels directly
[16:36:42] <awight>	 assuming this is a debian-stretch machine
[16:37:25] <tgr>	 I'm using a venv on stat1005
[16:37:57] <awight>	 You can check out mediawiki/services/ores/deploy.git to get the whole suite of production stuff, fwiw
[16:38:55] <awight>	 That dependency glitch is still surprising, though.
[16:39:29] <awight>	 Not that it *should* matter, but is this a fresh virtualenv or are you upgrading an existing one?
[16:40:10] <tgr>	 an existing one, it worked with an old version of ores
[16:41:22] <tgr>	 I ran into the same issue in the past though, that's why the vagrant role curled some requirements.txt in the past
[16:42:59] <awight>	 lol a google search for ContextualVersionConflict is the sadness
[16:43:48] <awight>	 I’d suggest deleting the virtualenv and recreating, and if there’s still an issue then I’m much more concerned
[17:00:43] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120819 (10awight) >>! In T181071#4119135, @mmodell wrote: >>>! In T181071...
[17:03:15] <wikibugs>	 (03PS1) 10Awight: Verbose logging for a deployment script [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425316
[17:04:07] <wikibugs>	 (03CR) 10Awight: [V: 032 C: 032] "self-merging log thing for diagnostics" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425316 (owner: 10Awight)
[17:10:11] * halfak runs to lunch
[17:10:14] <halfak>	 Oh codezee! 
[17:10:32] <halfak>	 I did some writing and pinged you on MediaWiki.  It's not what I wanted.  Didn't have the muse.  I'll try again today. 
[17:11:06] <halfak>	 Back in ~ an hour
[17:15:22] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120858 (10awight) @mmodell: We're running the fetch check with "bash -x"...
[17:16:37] <wikibugs>	 (03PS1) 10Awight: TEMPORARY: force failure in fetch_check, in order to cause output logging [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425319
[17:16:54] <wikibugs>	 (03CR) 10Awight: [V: 032 C: 032] TEMPORARY: force failure in fetch_check, in order to cause output logging [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425319 (owner: 10Awight)
[17:31:35] <wikibugs>	 (03PS1) 10Awight: Revert diagnostics [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425322
[17:33:50] <wikibugs>	 (03CR) 10Awight: [V: 032 C: 032] Revert diagnostics [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/425322 (owner: 10Awight)
[17:39:28] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4120939 (10awight) There's a wealth of surprising results on tin, /srv/dep...
[17:39:39] <awight>	 commuting.
[18:33:21] <codezee>	 halfak|Lunch: Saw your ping and went through it, I've archived the ACL draft and started on consolidating the old sigir one because its format is same as for cscw
[18:50:29] <halfak>	 codezee, gotcha.  Can you link me to that version?
[18:55:35] <codezee>	 halfak: going through it, we can put the initial part which talks about Wikipedia review in its own section, which can be immediately followed by the related works section, where the rest of it can go
[18:58:49] <codezee>	 or I'll rather let you complete it first maybe :P
[18:59:16] <halfak>	 I do want to take another stab at it. 
[18:59:26] <halfak>	 I'm also looking to write something else up. 
[18:59:55] <halfak>	 Oh yeah.  I want to dig into discussing what's cool about the WikiProject directory. 
[19:00:10] <halfak>	 I'll tackle that today and come back to NPP/AFC vs counter-vandalism tomorrow. 
[19:07:36] <codezee>	 halfak: I remember your thoughts on the WikiProjects directory regarding its flexibility, to help I can pen them down on the same page, then maybe you can alter it or add to it
[19:07:53] <halfak>	 Feel free to beat me to it. 
[19:08:05] <codezee>	 i'll add as a comment then
[19:08:17] <halfak>	 I want to develop a graphic that conveys the re-wireability of it. 
[19:11:53] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121339 (10mmodell) >>! In T181071#4120939, @awight wrote: > It would be g...
[19:12:28] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121341 (10mmodell)
[20:04:08] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10WMSE-Bug-Reporting-and-Translation-2018, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): The filters menu is closed when a filter colour is selected - https://phabricator.wikimedia.org/T188903#4121460 (10jmatazzoni)
[21:47:58] <awight>	 halfak: Thanks for the nudge to read “preserving margins”.  The last paragraph seems leveled at creating JADE, nicely planted seed!
[22:00:53] <srrodlund>	 Listening to the Mark Z testiomony about FB and he is talking a ton about AI
[22:02:06] <srrodlund>	 using it to judge good/bad content
[22:05:03] <awight>	 A Manichean worldview :-). Cool, thanks for the tip
[22:06:13] <srrodlund>	 Just interesting to listen to people talking about similar things from vastly different perspectives. This seems to be his go-to answer when the questions get tough
[22:06:39] <halfak>	 :D awight 
[22:06:56] <halfak>	 I really like Dr. Mugar.  I think you'll like him too.  I wonder if he'll be at CHI :) 
[22:21:37] <halfak>	 Next April 1st, we should announce our new mascot, the ORESy horsey.  🐎
[22:21:53] <halfak>	 🐴
[22:24:12] <Platonides>	 🧠
[22:34:24] <halfak>	 :) 
[22:36:27] * halfak runs away
[22:36:30] <halfak>	 have a good one folks!
[22:36:46] <awight>	 see ya
[23:03:25] <icinga-wm>	 PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert.
[23:03:48] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121947 (10awight) Back to the drawing board.  Including the packaged Pyth...
[23:04:25] <icinga-wm>	 RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting.
[23:04:55] <awight>	 Graphs are a bit strange, the request volume actually went down during a short flurry of errors.
[23:05:02] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121949 (10mmodell) @awight: why not build the virtualenv on a developer m...
[23:07:45] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121952 (10awight) @mmodell: That would be wonderful, but virtualenvs are...
[23:09:36] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121953 (10mmodell) So why even use virtualenv if they are based on site p...
[23:17:48] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4121994 (10awight) That's right, we do use the --system-site-packages flag...
[23:21:53] <halAFK>	 o/ awight 
[23:21:57] <halAFK>	 what's up?
[23:23:11] <awight>	 halAFK: not much
[23:23:26] <awight>	 I have a RL thing happening tho, have to watch a neighbor’s kid for 30 min maybe
[23:23:47] <halAFK>	 Is ORES OK?
[23:24:12] <awight>	 ah
[23:24:20] <awight>	 it was fine around the last alert
[23:24:24] <halAFK>	 Looks like our 500 rate is fluctuating. 
[23:24:24] <awight>	 checking again...
[23:24:29] <halAFK>	 https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=23&fullscreen&orgId=1
[23:24:54] <halAFK>	 Something happened 35 minutes ago. 
[23:24:58] <eddiegp>	 T191940
[23:24:59] <stashbot>	 T191940: Investigate 2018-04-10 global traffic drop - https://phabricator.wikimedia.org/T191940
[23:25:06] <eddiegp>	 That just happened.
[23:25:19] <halAFK>	 Thanks eddiegp.  Might be related. 
[23:25:24] <awight>	 sorry, gotta babysit for 30min now
[23:25:45] <awight>	 all the other graphs look reasonable, but I’d like to know what type of errors these were
[23:25:51] <awight>	 maybe MW API?
[23:25:52] <halAFK>	 500s
[23:25:58] <awight>	 I’d check that graph next
[23:26:09] <halAFK>	 They were an error so severe that we couldn't return anything else. 
[23:26:10] <eddiegp>	 Networking issues, basically everything from phabricator/gerrit to some wikis went down for a short period.
[23:26:20] <halAFK>	 Oh.  that could explain it. 
[23:26:27] <halAFK>	 I think changeprop hits us through varnish
[23:27:56] <halAFK>	 Confirmed looks like no 500s for a few minutes. 
[23:28:06] <halAFK>	 I'll let it be for now and check back in a little bit. 
[23:33:50] <awight>	 thanks for attending to it!
[23:34:11] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10WMSE-Bug-Reporting-and-Translation-2018, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): The filters menu is closed when a filter colour is selected - https://phabricator.wikimedia.org/T188903#4122022 (10Etonkovidova) 05Open>03Invalid The...