[00:04:58] halfak: when you get a chance, can you send me some literature about the doc2vec concept you talked about this morning? I am Interested. [00:05:46] Some of it: https://radimrehurek.com/gensim/models/doc2vec.html [00:05:59] I haven't written anything up yet about the idea. [00:06:01] :| [00:06:04] Will get on that [00:08:21] I wonder if we can use this approach for Marshall's project. I messaged him but he's on vacation until Wednesday. [00:37:30] 10ORES, 10Scoring-platform-team: Look at overhead of json codec and data passing to feature extraction workers - https://phabricator.wikimedia.org/T181809 (10Ladsgroup) @awight Can you elaborate more? Also is this an investigation or actually doing it? [00:37:52] 10ORES, 10Scoring-platform-team: Make it easier to keep ORES production and labs deployments in sync - https://phabricator.wikimedia.org/T181824 (10Ladsgroup) p:05Triage>03Low [00:41:31] 10ORES, 10Scoring-platform-team: Spike in ORES errors every 24hr - https://phabricator.wikimedia.org/T191864 (10Ladsgroup) p:05Triage>03Lowest [00:47:39] 10ORES, 10Scoring-platform-team, 10Icinga, 10monitoring: ORES worker icinga message not specific enough - https://phabricator.wikimedia.org/T181536 (10Ladsgroup) p:05Triage>03Lowest Actually, we know send a header that which ores node served the request but that would be the uwsgi node and not the work... [00:49:23] 10ORES, 10Scoring-platform-team, 10Scap: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10Ladsgroup) This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? [00:54:39] 10ORES, 10Scoring-platform-team, 10Scap: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) >>! In T182258#4776389, @Ladsgroup wrote: > This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? Sure! Addition... [00:54:44] 10ORES, 10Scoring-platform-team, 10Scap: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) 05Open>03Resolved a:03awight [01:03:21] 10ORES, 10Scoring-platform-team: Look at overhead of json codec and data passing to feature extraction workers - https://phabricator.wikimedia.org/T181809 (10awight) >>! In T181809#4776373, @Ladsgroup wrote: > @awight Can you elaborate more? Also is this an investigation or actually doing it? I was imagining... [01:11:56] 10ORES, 10Scoring-platform-team: Look at overhead of json codec and data passing to feature extraction workers - https://phabricator.wikimedia.org/T181809 (10awight) [01:14:43] 10Scoring-platform-team, 10monitoring: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10Ladsgroup) [01:14:45] 10ORES, 10Scoring-platform-team, 10monitoring, 10Wikimedia-Incident: Create Grafana graph to show number of ORES API requests per user-agent - https://phabricator.wikimedia.org/T182222 (10Ladsgroup) [01:16:04] 10Scoring-platform-team, 10monitoring: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10Ladsgroup) p:05Triage>03Normal Let's do it in logstash by sending the proper json from uwsgi to logstash. I can work on it in some time. [01:17:21] (03PS1) 10Awight: [WIP] Summarize preferred judgment values in link table [extensions/JADE] - 10https://gerrit.wikimedia.org/r/475932 (https://phabricator.wikimedia.org/T200297) [01:19:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Summarize preferred judgment values in link table [extensions/JADE] - 10https://gerrit.wikimedia.org/r/475932 (https://phabricator.wikimedia.org/T200297) (owner: 10Awight) [01:22:37] 10Scoring-platform-team, 10monitoring: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10awight) @Ladsgroup Do you have thoughts about the privacy issues? I don't think we release raw user-agent any more, although there's some historical precedent [... [01:22:44] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Summarize preferred judgment values in link table [extensions/JADE] - 10https://gerrit.wikimedia.org/r/475932 (https://phabricator.wikimedia.org/T200297) (owner: 10Awight) [01:34:24] 10ORES, 10Scoring-platform-team: Jinja error in ORES - https://phabricator.wikimedia.org/T183949 (10awight) 05Open>03Invalid I checked on ores1001 and locally, and "unicode-escape" is a valid encoding. Let's call this "invalid" until it happens again. [01:39:18] 10ORES, 10Scoring-platform-team: Make Celery `result_backend` and `broker_url` configurable by environment variable - https://phabricator.wikimedia.org/T195074 (10awight) The details are vague to me at this point, but I think the issue was that we would be spinning up several docker containers and they have to... [01:43:45] Amir1: Awesome work on the backlog! [01:45:58] awight: thank you for picking my work up. I'm just cleaning up a little bit to pick my next batch of work on maintenance [01:46:15] :) we needed this [01:46:29] I shamelessly only cleaned up the maintenance column [01:56:12] haha good to set expectations [02:06:08] fleeing. [06:34:55] 10Scoring-platform-team, 10monitoring: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10Ladsgroup) >>! In T181542#4776461, @awight wrote: > @Ladsgroup Do you have thoughts about the privacy issues? I don't think we release raw user-agent any more,... [11:36:20] (03PS1) 10Ladsgroup: Bump ORES to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/475993 [11:37:14] (03CR) 10Ladsgroup: [V: 032 C: 032] "Deploying to beta" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/475993 (owner: 10Ladsgroup) [15:25:59] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) The second bullet point ("filter RC for edits which are ORES-nondamaging and JADE-damaging") seems l... [15:37:59] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 6471 bytes in 5.048 second response time [15:43:20] Uh oh [15:43:40] I was able to get a score through. [15:44:24] icinga, explain. [15:44:49] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 923 bytes in 0.097 second response time [15:45:26] https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=23&fullscreen&orgId=1&from=1543331330609&to=1543332606101 [15:45:30] Looks like there was a blip. [15:45:35] akosiaris, ^ [15:45:48] 15:21 UTC, we had a spike in 500 responses. [15:46:11] 0.040 ? [15:47:04] I don't see anything else that is concerning in the graphs [15:47:35] yeah. That's a bigger spike than you might imagine. But still, only 4% of requests. [15:48:18] it's not a % [15:48:29] 0.040 reqs per min [15:48:33] at the peak [15:49:13] Ahh yeah. [15:49:32] We had a bigger spike at 0830 [15:49:37] https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=23&fullscreen&orgId=1&from=1543305677350&to=1543311104436 [15:50:28] the "scores errored" does show some small spikes, but it's like 6 per min at the highest [15:50:42] These 500s shouldn't be happening at all. [15:50:53] I would look at logstash for error logs. INTERNAL SERVER ERROR sounds like something that is logged [15:50:54] Maybe it was mwapi. I keep forgetting that we can get 500s from mwapi fails. [15:51:04] Good point. [15:52:40] mwapi.errors.APIError: internal_api_error_MediaWiki\Revision\RevisionAccessException: [W-1gawpAMDgAAKQ6ToUAAABR] Caught exception of type MediaWiki\Revision\RevisionAccessException -- None [15:52:48] And a few API timeouts. [15:53:06] Is there a good way to check on mwapi metrics? [15:54:18] https://grafana.wikimedia.org/dashboard/db/api-backend-summary?orgId=1&from=1543322738637&to=1543334011705 [15:54:26] Something might be happening around that time. [16:35:20] akosiaris: halfak This is log of all 500 errors in the last 24 hours: https://logstash.wikimedia.org/goto/7bd46fd52b86553f93f1c1e011ee7f2d [16:35:43] random connection failed error [16:37:15] yeah spiky S1 at times [16:37:24] DBAs are complaining a bit as well [16:37:28] something minor is up [16:37:33] still investigating though [16:38:42] https://phabricator.wikimedia.org/T210499 [16:38:47] if you want to follow up [16:39:50] is there a way to search for all log items that do not contain "mwapi.errors.TimeoutError"? [16:39:53] I'm struggling [16:40:13] * halfak prepares for meetings [16:41:48] halfak: Sure: https://logstash.wikimedia.org/goto/63096e5b35731ae970f8df9068fb4905 [17:27:01] How strange, wikibugs is taking a little break. [17:27:15] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4777873, @Halfak wrote: > The second bullet point ("filter RC for edits which are ORE... [17:27:37] O_O [17:27:46] well then [17:27:55] I need to relocate, back in 15min [17:59:10] Thanks Pchelolo :) [17:59:55] I wanna make KSQL emit a stream of 'best editors according to ORES' with a single query [18:00:20] just to check how ksql could work on this kind of data [18:03:13] Pchelolo: FYI, there is a cache of recent ORES scores in the MediaWiki database, if that make your join any easier. [18:03:54] Pchelolo: Also, please consider whether "non-damaging" or "goodfaith" is a better metric for what you want to see in an editor. Or whether the "articlequality" increased due to their edits. [18:04:07] awight: nono, the whole point is to try the kafka KSQL thingy [18:04:16] hehe [18:04:41] Not the easy way, then ;-) [18:04:42] it's just a fun project to check out new tech [18:04:56] Here's a related experiment you might like to read about, https://meta.wikimedia.org/wiki/Research:ORES-powered_TeaHouse_Invites [18:05:18] kk makes sense to me, glad ORES might be useful! [18:05:30] cool. thank you. this is more info then I was hoping to get :) [18:05:45] I'll get back to you if my little experiment is successfull [18:06:38] Pchelolo: oh of course! I think I get it now--you're reading from the kafka ORES stream, that's exciting! [18:08:28] 10Scoring-platform-team, 10monitoring: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10awight) >>! In T181542#4776713, @Ladsgroup wrote: > So the plan is to have the data in logstash which is already behind NDA. Thanks for pointing that out! [18:46:22] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) >>! In T200297#4778378, @awight wrote: >>>! In T200297#4777873, @Halfak wrote: > Harej and I chatted... [19:03:02] awight, docs meeting? [19:04:20] headed [19:29:21] 10JADE, 10Scoring-platform-team, 10User-Testing: [Epic] Code support for Jade user testing - https://phabricator.wikimedia.org/T210535 (10awight) [19:31:28] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4778731, @daniel wrote: >>>! In T200297#4778378, @awight wrote: >>>>! In T200297#4777... [19:36:00] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > I think we can support filtering by adding an index on the summary data? I was planning to do this... [19:44:54] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4778966, @daniel wrote: >> I think we can support filtering by adding an index on the... [19:55:33] 10JADE, 10Scoring-platform-team, 10User-Testing: [Epic] Code support for Jade user testing - https://phabricator.wikimedia.org/T210535 (10Harej) [19:55:36] 10JADE, 10Scoring-platform-team (Current), 10Design, 10Patch-For-Review: Come up with view mode for JADE pages - https://phabricator.wikimedia.org/T208819 (10Harej) [20:00:52] PROBLEM - ping4 on Experimental ORES Website is CRITICAL: CRITICAL - Host Unreachable (ores.wmflabs.org) [20:01:05] PROBLEM - Host Experimental ORES Website is DOWN: CRITICAL - Host Unreachable (ores.wmflabs.org) [20:01:13] big deal. [20:01:54] ores.wmflabs.org doesn't look down to me. [20:02:16] icinga2-wm_: meet icinga-wm [20:02:31] * awight is disappointed in lack of botwar [20:10:51] RECOVERY - Host Experimental ORES Website is UP: PING OK - Packet loss = 0%, RTA = 1.09 ms [20:12:53] Looks like we need to build an image recognition algorithm for vandal fighting. https://www.theverge.com/2018/11/24/18110545/wikipedias-trump-penis-vandalism-account-hacking [20:14:57] halfak: have you decided yet that doc2vec is something you definitely want to use, or are you more exploring it at this point? [20:15:22] harej, exploring. I'd like to reach out to some of my embeddings-crazy collaborators for advice at some point. [20:15:59] But for the time being, I think we can assume that will have a "good" embedding and work from there. [20:17:02] embedding? [20:21:15] "Chunk of text" --> "Embedding" --> [vector of numbers] [20:21:29] harej, ^ [20:21:40] I see. [20:21:54] Basically the embedding "embeds" high signal information about stuff in a (usually highly dimensional) space. [20:22:01] Like long vectors. [20:24:06] halfak: very nice product placement for 2FA. What on Earth are admins doing without that... [20:24:31] No idea. I've been speaking the good word as often as I can. [20:25:10] I administered a friend's G Suite stuff for a while and it was insanely difficult to get *four* people to all enable 2FA [20:26:46] Even enforcing 2FA-only wasn't enough, because they just wouldn't use the account in that case. [20:27:09] on that note, I'm gonna forage for nuts and berries. biab [20:35:56] 10JADE, 10Scoring-platform-team, 10User-Testing: [Epic] Code support for Jade user testing - https://phabricator.wikimedia.org/T210535 (10Harej) [20:36:00] 10JADE, 10Scoring-platform-team (Current), 10Design: Discuss and create a UI mockup for the JADE editor interface - https://phabricator.wikimedia.org/T168993 (10Harej) [20:38:44] awight: https://phabricator.wikimedia.org/T199128 << is this a subtask of https://phabricator.wikimedia.org/T210535 ? [20:42:04] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10JAllemandou) Another comment about folders that I hadn't thought before having read your update in the description: I actually think that the chosen is not the most... [20:45:00] harej: Thanks! [20:45:09] I was asking! [20:45:10] I should go ahead and define some more subtasks, thanks for nudging [20:45:40] My hesitation is that it was ranked as lower priority, but I have a feeling we have mentally given it higher priority, unless this task is for a shiny bauble we don't need right away. [20:47:32] harej: ah I thought you had linked and that's why I was getting notifications. As originally written, I meant for /T199128 to be the "real" view UI, but yeah let's scope this to be "good enough for testing" rather than real, necessarily. [20:47:53] +1 good call [20:48:08] everything, for that matter. [20:49:25] 10JADE, 10Scoring-platform-team, 10Design: Create overlay UI for editing Judgement pages - https://phabricator.wikimedia.org/T199128 (10Harej) p:05Low>03Normal [20:49:36] 10JADE, 10Scoring-platform-team, 10Design: Create overlay UI for editing Judgement pages - https://phabricator.wikimedia.org/T199128 (10Harej) [20:49:38] 10JADE, 10Scoring-platform-team, 10User-Testing: [Epic] Code support for Jade user testing - https://phabricator.wikimedia.org/T210535 (10Harej) [21:03:03] harej: "no opinion was given as to whether the UI paradigm is good or not" <-- +1, We need to find out what people think [21:24:33] https://meta.wikimedia.org/wiki/User:Halfak_(WMF)/Trip_reports/Berkman_AI_%26_Ethics_2018 [21:24:35] Bam [21:33:39] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > There's no prefix matching though, these are two tinyint fields holding a boolean each. Ah right... [21:35:07] harej: I think we're on the right track here but would like to hear what you think about the filtering use case: https://phabricator.wikimedia.org/T200297#4779339 [21:40:13] I'm going to skip writing about PAI. I just don't know what to say that isn't sad and negative. [21:40:31] So that means I'm working on Italian edit quality models. [21:40:46] Maybe I should rebuild the wikidata models using the new mwbase. [21:41:05] o/ hoo [21:41:10] hi halfak :) [21:41:19] I'm wondering where you are with the PropertySuggestion features. [21:41:28] It would be good timing if we were ready to experiment with them now :) [21:45:56] It's always good timing to have a good thing :p [21:59:11] hoo ^ [22:02:44] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Thanks, this has been a helpful tangent! >>! In T200297#4779339, @daniel wrote: >> If you feel that... [22:04:05] FML, just lost a phab tab with a ton of edits [22:04:45] aah whew, it was shipwrecked in a detached window. [22:07:19] Oh nice. [22:11:13] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > There should be no impact on revision pager queries until we have UI to filter on the new index.... [22:19:36] 10JADE, 10Scoring-platform-team, 10Design: Create overlay UI for editing Judgement pages - https://phabricator.wikimedia.org/T199128 (10awight) [22:20:24] halfak: I think you have a typo in your report. "The session I as a Wikimedia representative gathered quite a bit of interest." [22:21:01] edits are welcome if you feel up to it. :) [22:22:35] 10JADE, 10Scoring-platform-team, 10Design: Create overlay UI for editing Judgement pages - https://phabricator.wikimedia.org/T199128 (10awight) [22:24:12] Well I'm not sure what you meant to say. The session you presented? Attended? [22:29:19] ah verbing [22:29:31] "led" perhaps [22:29:42] * awight looks again [22:30:15] yeah that would be led [22:32:23] I don't know what I meant to say either. [22:32:25] :D [22:33:02] * awight imagines halfak not knowing what to say during the workshop he. [22:33:36] That's an inspiring trip report, btw! I really need to digest the notes I took at CSCW. [22:35:05] lol [22:35:19] I really liked the Berkman event. I hope to work with them more in the future. [22:35:36] The legal folks really like them too, so I suspect that will be happening :D [22:35:56] OK I'm out of here. Time to go bike in place and then pick things up and put them down. :) [22:35:58] o/ [22:38:22] o/ fill in those holes [23:03:07] harej: Did you and Aaron ever work out the question of "is null" for damaging or goodfaith? [23:04:45] Anyway, having T209524 written as a task is all I need for now. [23:04:46] T209524: In a multi-part judgment, can one of the parts be null? - https://phabricator.wikimedia.org/T209524 [23:05:41] 10JADE: In a multi-part judgment, can one of the parts be null? - https://phabricator.wikimedia.org/T209524 (10awight) @Halfak We need your feedback here. All I remember from IRC was that you wanted "not null" validation, but don't know what your reasoning is. [23:06:50] 10JADE, 10Scoring-platform-team, 10User-Testing: Prototype Jade controls on the action=diff page - https://phabricator.wikimedia.org/T210558 (10awight) [23:09:06] harej: Should we be testing watchlist integration? I'm thinking yes. [23:12:47] (made a note in the test doc) [23:13:47] i don't see a task for the Watchlist integration, stubbing something... [23:20:02] 10JADE, 10Scoring-platform-team, 10User-Testing: Design and implement prototype of Jade integration with Special:Watchlist - https://phabricator.wikimedia.org/T210560 (10awight) [23:27:13] 10JADE, 10Scoring-platform-team, 10User-Testing: Create test fixtures for Jade user testing - https://phabricator.wikimedia.org/T210562 (10awight) [23:27:43] harej: That's all I can think of for testing tasks, but I might be leaving out something obvious. [23:46:21] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4779180, @JAllemandou wrote: > Another comment about folders that I hadn't thought before having read your update in the description: I actual... [23:57:56] awight: Aaron and I never had that argument; should we put it on the docket for staff meeting? [23:58:43] and yes, we should test watchlist integration. Namely, is it a good idea to filter RC entries based on the existence of a Jade page for that diff [23:59:40] can you add that to the test doc?