[00:35:01] noo. I realized that score events will include redundancy between datacenters, so we have to merge both streams and filter to distinct rows. [05:54:40] halfak https://www.technologyreview.com/s/612726/this-algorithm-browses-wikipedia-to-auto-generate-textbooks/ [08:09:19] 10Scoring-platform-team, 10Serbian-Sites: Investigate srwiki goodfaith model, why is it so bad? - https://phabricator.wikimedia.org/T199355 (10Liuxinyu970226) [08:09:48] 10ORES, 10Scoring-platform-team, 10editquality-modeling, 10Serbian-Sites, 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870 (10Liuxinyu970226) [09:01:03] Late nite ramble: our AI services fall under at least two categories: triage (finding prioritized needles in the info overload haystack) and sorting (article quality, draft topic, putting things in different buckets for hierarchy and not for prioritizing) [09:04:47] In pursuing business opportunities, there are (at least) two questions to answer: where can we highlight priorities among an excess of data? And: how can we make it easier to sort piles things into buckets? [09:09:42] They’re different questions in my opinion but what they have in common is: how do we find our way through the big and chaotic city? When there are billions of restaurants, which ones do you stop at? Where are the different neighborhoods? Which ones have the best whiskey collection? (Lagavulin or bust). What are the articles on Wikipedia about whiskey? What about the ones adjacent to whiskey? How do I do this on a wiki large enough to have more than [09:09:42] 100 articles but not necessarily so large as to have an extremely robust category system? [09:10:58] unpopular opinion: there are no wikis with extremely robust category systems [09:26:49] I guess a better word to reflect my intention would be “comprehensive” [09:27:37] English Wikipedia and Commons categories: very comprehensive. Robust, doubtful. [09:28:16] Anyways, reminder to self to make my ramble into a proper office wiki essay [10:11:19] o/ [10:18:52] heh [10:22:53] true story about en wikipedia categories: https://twitter.com/jesswade/status/1073167328890761216 [10:22:58] (see the thread) [10:23:12] that was ONE MONTH ago *cough* [15:31:01] Well that was mostly painless. [16:19:02] 10ORES, 10Scoring-platform-team, 10translatewiki.net, 10Security: New ORES model relies on translatewiki.net API, which is not hosted on WMF production - https://phabricator.wikimedia.org/T213131 (10Nemo_bis) [17:01:24] I'm heading into the University so I'll be offline for ~45 minutes. [18:10:20] Well that took a bit longer than expected. [18:31:47] 10Jade, 10Scoring-platform-team, 10Design: Jade Wireframes: Entity view mode - https://phabricator.wikimedia.org/T212379 (10Halfak) I'm not a designer, but I figured that it would help jump-start our design conversation if I took a stab at some layouts. Below are three wireframes that I have worked out to d... [18:39:12] O_O why did scores processed just drop to zero [18:40:52] the heck is happening [18:43:12] Amir1: halfak: grafana has been glitching for me. Anyone else available to take a quick peek and confirm that we're serving *zero* requests? [18:43:15] I don't buy it. [18:44:29] awight: change the timespan [18:44:51] it's a glitch in grafan, turn it from the last two hours to three hours or one hour. [18:45:23] I still see a drop at 18:36 [18:46:00] There's a MW deployment happening, but it didn't start until 18:43 [18:47:14] What I'm seeing is that precache requests dropped by 70% on eqiad and 100% on codfw [18:47:35] so... I guess it's not us [18:47:37] that's probably some real issue then [18:47:54] :) [18:48:06] thanks--as long as I'm not going crazy. [19:36:57] 10ORES, 10Scoring-platform-team, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Backlog): The continuous release pipeline should support more than one service per repo - https://phabricator.wikimedia.org/T210267 (10thcipriani) I think there are a couple of problems with the curre... [19:49:51] (03PS1) 10Umherirrender: Use RecentChange::getAttribute instead of property access [extensions/JADE] - 10https://gerrit.wikimedia.org/r/483839 [20:36:08] 10ORES, 10Scoring-platform-team, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Backlog): The continuous release pipeline should support more than one service per repo - https://phabricator.wikimedia.org/T210267 (10Ottomata) Seems like it would work, but it doesn't look like this... [20:49:17] o/ awight [20:49:29] how did the run go with Claudia's group? [20:53:18] 10Scoring-platform-team, 10Analytics: Investigate formal test framework for Oozie jobs - https://phabricator.wikimedia.org/T213496 (10greg) removing #releng as this seems like just an investigation task, please do let us know what you find out! [20:55:28] halfak: Still working on it, something odd is happening. [20:55:41] Oh. Anything I can help you think about? [20:55:53] The parallelism was great at first, and now requests are barely trickling through. [20:56:04] It could be that problem I was talking about. [20:56:07] halfak: yeah--do you know how we forward requests for esams? [20:56:14] s/for/through [20:56:18] esams? [20:56:22] halfak: it certainly could be! [20:56:28] I don't think requests even hit esams [20:56:38] Yeah, ores.wikimedia.org resolves to the euro datacenter it seems, but there's no ORES there. [20:56:47] Huh. no idea then. [20:56:57] same here. will ask in -operations [20:57:01] But I do think that concurrent.futures.ThreadPool is finicky. [20:57:15] I'll submit a PR that I think might deal with the issue. [21:37:23] 10ORES, 10Scoring-platform-team (Current): ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10awight) a:03awight Hi, I'm adopting this bug because it seems related to a recent glitch we discovered today. A researcher using the ORES Python client with a parallelism of 20... [21:38:00] halfak: I'll check out your PR, and meanwhile have some other thoughts about our socket management. [21:38:18] It seems that FU-Berlin was wallowing in TIME_WAIT sockets [21:38:39] AFAIK, that means we aren't closing the connection properly in some cases. [21:38:44] Either way, we need a good way to solve for the problem I'm looking at. [21:38:48] Gotcha. [21:39:27] +1, if you already have a PR I can review, or I can try to implement. [21:40:41] yeah looking more carefully, your issue is something else. I'll make a new task [21:41:12] 10ORES, 10Scoring-platform-team (Current): ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10awight) a:05awight→03None [21:42:15] 10ORES, 10Scoring-platform-team (Current): ORES Python client drowning in TIME_WAIT sockets - https://phabricator.wikimedia.org/T213582 (10awight) [21:43:56] 10ORES, 10Scoring-platform-team (Current): ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10awight) I think my last comment is a different bug, so splitting into T213582 [21:52:10] * halfak works on making this error scenario testable. [21:56:36] halfak: Did you have a reason to use requests.get stream=True? [21:56:42] I think that's the entire issue... [21:57:00] Gotcha. Could be indeed. I can't tell you why that exists. [21:57:03] From the docs, it seems that stream is only useful if we'll be canceling the request before downloading the entire content [21:57:06] kk nice [21:57:11] I'll remove it... [22:03:52] wikimedia/ores#1239 (no_stream - 037b28e : Adam Wight): The build passed. https://travis-ci.org/wikimedia/ores/builds/478554914 [22:07:01] awight, added a comment. [22:07:15] Not sure why you put the request in the try. [22:07:37] Once we get this merged, we can rebase and you'll get to see what I worked out re. tests. [22:13:57] I think my work will be critical to the FU-berlin processing. [22:14:30] As it stands, if an API call errors out, the whole script should crash. I've made it so that an error will be reported and outputted in place of a score instead. [22:14:56] reported --> Logged. [22:16:18] haha, parallel programming is hard. I just discovered that their client script is doing its own batching, which will cause our threads to all wait for the slowest request out of each 1,000 scores. [22:16:40] :) That's a great change, nice one! [22:19:41] Ooh. Barrier synchronization. [22:22:37] awight, https://github.com/wikimedia/ores/pull/309 [22:26:02] wikimedia/ores#1244 (score_revisions_handling - 0a36f67 : halfak): The build failed. https://travis-ci.org/wikimedia/ores/builds/478563652 [22:26:32] No [22:26:34] Why [22:26:36] :'( [22:27:25] awight, none of those errors have anything to do with my pr. [22:27:31] They are all in test_server [22:27:42] recheck? [22:28:12] ./ores/tests/test_api.py:1:1: F401 'pytest' imported but unused [22:29:00] Bah nevermind [22:29:05] Yeah. Getting that [22:29:39] OK fixed. [22:30:00] awight, I'm curious what you think of the mock. [22:32:49] Hello. I think this might be the best place. While compiling a puppet diff for an unrelated issue I found an ORES host not compiling and details are at https://puppet-compiler.wmflabs.org/compiler1002/94/orespoolcounter1002.eqiad.wmnet/change.orespoolcounter1002.eqiad.wmnet.err [22:32:54] halfak: Did you add session= for testing only, or do you plan to reuse sessions? [22:33:16] wikimedia/ores#1246 (score_revisions_handling - a81508d : halfak): The build was fixed. https://travis-ci.org/wikimedia/ores/builds/478566002 [22:33:35] awight, testing only. Might be useful otherwise? Not sure. [22:34:08] thanks Hauskatze [22:34:20] np :) [22:34:28] Not sure what to do with that puppet error, but I'll turn it into a task :) [22:34:56] a bird told me that one of the causes could be something added in a private repo and no fake values added elsewhere [22:35:19] that said, Puppet is still very very hard to understand to me [22:35:43] 10ORES, 10Scoring-platform-team (Current), 10Puppet: orespoolcounter1002.eqiad.wmnet reporting compile errors - https://phabricator.wikimedia.org/T213586 (10Halfak) [22:36:35] 10ORES, 10Scoring-platform-team (Current), 10Puppet: orespoolcounter1002.eqiad.wmnet reporting compile errors - https://phabricator.wikimedia.org/T213586 (10Halfak) Thanks to @MarcoAurelio for reporting. I don't have enough puppet expertise to know what to do with this. But I figure that @akosiaris might k... [22:36:54] Hauskatze, yeah I'm in the same boat. :) [22:39:20] wikimedia/ores#1244 (score_revisions_handling - 0a36f67 : halfak): The build has errored. https://travis-ci.org/wikimedia/ores/builds/478563652 [22:39:31] wat. You just said you were fixed. [22:40:02] "All checks have passed" https://github.com/wikimedia/ores/pull/309 [22:40:08] Travis, you're drunk [22:40:20] AsimovBot, take travis home [22:40:20] 04Error: Command “take” not recognized. Please review and correct what you’ve written. [22:47:44] xD [22:47:52] they're both either drunk or high on drugs [22:49:51] I'm going to get on my bike to avoid the darkness (mostly). Awight, I imagine you have what you need to support the FU-berlin folks. Feel free to take over my PR and ping me for a quick +2 if you need it. Gchat should be best [22:49:52] o/ [22:53:33] halfak: May it be a nice ride. This morning's sounded harrowing, or long at least. [23:24:33] awight: you apply to go to the hackathon? [23:25:13] harej: Not yet, thanks for the nudge! [23:25:27] * awight leaps on the ticking clock [23:26:39] is the central notice banner displaying correctly for y'all? [23:45:33] wikimedia/ores#1248 (plain_input - 2104668 : Adam Wight): The build has errored. https://travis-ci.org/wikimedia/ores/builds/478592436 [23:48:04] wikimedia/ores#1249 (plain_input - 194c0e2 : Adam Wight): The build passed. https://travis-ci.org/wikimedia/ores/builds/478592625