[02:44:47] 06Revision-Scoring-As-A-Service, 10ORES: Have a metric for direct requests to ORES not scoring ones - https://phabricator.wikimedia.org/T139446#2494616 (10Ladsgroup) 05Open>03Invalid [02:46:25] 06Revision-Scoring-As-A-Service, 10ORES: Have a metric for direct requests to ORES not scoring ones - https://phabricator.wikimedia.org/T139446#2432568 (10Ladsgroup) We do have a metric for hitting cache (which I was unaware of) we just summed it with scoring requests and it worked. E.g total number of request... [03:36:26] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:16] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 339 bytes in 0.692 second response time [14:18:01] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Spec] Deployment tests for ORES - https://phabricator.wikimedia.org/T140412#2495664 (10Halfak) [14:20:15] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Add description when a user reviews an edit - https://phabricator.wikimedia.org/T136903#2351790 (10Halfak) This appears to be done and deployed. [14:20:26] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Add description when a user reviews an edit - https://phabricator.wikimedia.org/T136903#2495683 (10Halfak) [14:20:43] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Add description when a user reviews an edit - https://phabricator.wikimedia.org/T136903#2351790 (10Halfak) a:03Ladsgroup [14:20:48] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Spec] Deployment tests for ORES - https://phabricator.wikimedia.org/T140412#2463824 (10Ladsgroup) #scap3 has some utilities that can run some tests especially tests that hit API node and check if they are healthy. [14:23:37] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Add a new grafana pane for timeout error metric - https://phabricator.wikimedia.org/T140363#2495694 (10Halfak) It looks like the timeout error metric still needs to be deployed. @Ladsgroup, should this have been caught in the last deployment? [14:23:43] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Add description when a user reviews an edit - https://phabricator.wikimedia.org/T136903#2495696 (10Ladsgroup) @Halfak, I don't think so. I think it meant if someone check out a diff page ([[ https://en.wikipedia.org/w/index.php?title=United_States_H... [14:25:09] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Add a new grafana pane for timeout error metric - https://phabricator.wikimedia.org/T140363#2495697 (10Ladsgroup) I actually just deployed it last night and I'm checking graphite to see if any thing gets caught up but until now nothing happened (and that's kind... [14:25:55] o/ halfak [14:26:10] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Add "info" URL to campaign data so that we can link to campaign page - https://phabricator.wikimedia.org/T139957#2495698 (10Halfak) a:03schana [14:26:24] 06Revision-Scoring-As-A-Service, 10Wikilabels: Add "info" URL to campaign data so that we can link to campaign page - https://phabricator.wikimedia.org/T139957#2448002 (10Halfak) [14:26:52] o/ Amir1 [14:28:12] hey, I'm trying to see why time out errors haven't made into graphite [14:29:19] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Wikilabels UI reports non-200 status errors badly - https://phabricator.wikimedia.org/T138255#2495708 (10Halfak) Sometimes the "fail()" condition of an AJAX request is a 409 or 50X error that contains well formed JSON. In those cases, we should report the... [14:29:34] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Wikilabels UI reports non-200 status errors badly - https://phabricator.wikimedia.org/T138255#2495709 (10Halfak) a:03schana [14:30:16] http://bit.ly/2atUrW4 [14:30:30] halfak: ^ this is the link to logstash (it's super long) [14:30:50] "Model contains an error: The operation timed out." [14:32:21] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Include specific user groups in the trwiki edit quality model - https://phabricator.wikimedia.org/T140474#2495724 (10Halfak) [14:33:40] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Include specific user groups in the trwiki edit quality model - https://phabricator.wikimedia.org/T140474#2466127 (10Halfak) a:03schana [14:35:15] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10revscoring, 07Documentation: Add MacOS instructions for installation to README - https://phabricator.wikimedia.org/T139355#2495729 (10Halfak) a:03schana [14:37:11] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Investigate 20s long gaps in ORES logs - https://phabricator.wikimedia.org/T117825#2495732 (10Halfak) 05Open>03declined [14:37:23] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Investigate 20s long gaps in ORES logs - https://phabricator.wikimedia.org/T117825#1784389 (10Halfak) Declining because this is very old and no longer relevant. [14:40:41] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-articlequality : NLP for article quality models. - https://phabricator.wikimedia.org/T132533#2495748 (10Halfak) [14:40:57] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-articlequality : NLP for article quality models. - https://phabricator.wikimedia.org/T132533#2201702 (10Halfak) a:03schana [14:46:04] * halfak clicks on logstash link [14:46:26] Amir1, does it look like this is happening too often? [14:46:40] 1% [14:46:54] OK. That's pretty often, but not crazy. [14:47:06] Could these logs include the rev_id being scored? [14:47:10] not bad, but I want this to be logged somewhere to find solutions [14:47:11] It would be nice to review them. [14:47:20] nope, I tried that :( [14:47:32] I can add it but it'll take some time [14:47:45] Let's make a task for investigating this. [14:47:47] We need to deploy it, etc. [14:48:00] Indeed. [14:48:07] I think that' [14:48:11] s a good idea long term [14:48:19] We'll want enough information to reproduce an error. [14:48:34] Maybe the full URL of the request could be included. [14:49:19] yeah [14:49:28] I'll do it [14:52:11] halfak: https://etherpad.wikimedia.org/p/ores_weekly_update [14:52:23] Nice [14:52:24] It would be great if yo review it and then I add links and stuff [14:54:49] Amir1, OK looks good [14:54:55] Made some minor modifications [14:54:56] Thanks :) [14:54:57] thanks :) [14:57:32] halfak: btw. https://grafana.wikimedia.org/dashboard/db/ores I added some links at the top. Check them and modify/remove/add anything you want [14:57:56] Looks good. [15:01:31] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Set up grafana dashboard for ORES extension - https://phabricator.wikimedia.org/T141169#2495791 (10Ladsgroup) 05Open>03Resolved [15:01:34] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Set up "canary" node for ORES in prod - https://phabricator.wikimedia.org/T139825#2495792 (10Ladsgroup) 05Open>03Resolved [15:01:38] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10Wikidata, 13Patch-For-Review, and 3 others: ORES extension score only main namespace edits for Wikidata - https://phabricator.wikimedia.org/T139660#2495793 (10Ladsgroup) 05Open>03Resolved [15:01:41] 06Revision-Scoring-As-A-Service, 10ORES, 07Easy: "Hosted in Wikimedia Labs" is wrong ;) and provide a link to source code in github - https://phabricator.wikimedia.org/T139028#2495794 (10Ladsgroup) 05Open>03Resolved [15:01:43] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-07-26_(1.28.0-wmf.12): More robust way of handling errors in Cache.php - https://phabricator.wikimedia.org/T137880#2495795 (10Ladsgroup) 05Open>03Resolved [15:01:46] OMG SPAM [15:02:11] five tasks only :( [15:02:16] we had 23 once [15:02:49] Between each crest is a trough. [15:03:29] :) [15:03:46] halfak: have you made the task for investigating time out errors in log? [15:03:52] (so I don't make another one) [15:03:56] negative. [15:04:23] okay, I just make one [15:08:24] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-log-errors: [Investigate] ORES time out errors in logs - https://phabricator.wikimedia.org/T141368#2495817 (10Ladsgroup) [15:11:19] (03PS1) 10Ladsgroup: Add revision_id to log for errors [extensions/ORES] (wmf/1.28.0-wmf.11) - 10https://gerrit.wikimedia.org/r/301135 (https://phabricator.wikimedia.org/T141368) [15:13:11] (03Abandoned) 10Ladsgroup: Add revision_id to log for errors [extensions/ORES] (wmf/1.28.0-wmf.11) - 10https://gerrit.wikimedia.org/r/301135 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [15:14:29] (03PS1) 10Ladsgroup: Add revision_id to log for errors [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301136 (https://phabricator.wikimedia.org/T141368) [16:09:39] legoktm: around? and have some free time? [16:29:08] * halfak pushes ores 0.8.0 to pypi [16:29:10] \o/ [16:40:36] Amir1, https://github.com/wiki-ai/ores-wmflabs-deploy/pull/66 [16:41:45] nice [16:42:03] 06Revision-Scoring-As-A-Service, 10ORES: precached: Merge multiple models per rev_id into a single request - https://phabricator.wikimedia.org/T141376#2496102 (10Halfak) [16:42:46] 06Revision-Scoring-As-A-Service, 10ORES: Update wmflabs deploy repo for new version of ORES - https://phabricator.wikimedia.org/T141377#2496119 (10Halfak) [16:42:52] 06Revision-Scoring-As-A-Service, 10ORES: Update wmflabs deploy repo for new version of ORES - https://phabricator.wikimedia.org/T141377#2496133 (10Halfak) https://github.com/wiki-ai/ores-wmflabs-deploy/pull/66 [16:43:20] OK, I'm away for lunch. When I get back, I'll be working on communication and documentation for the review tool. [16:44:41] I'm here writing some tests for the extension. I'll probably go home soon [16:54:09] (03PS1) 10Ladsgroup: First CI test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) [17:02:28] (03CR) 10jenkins-bot: [V: 04-1] First CI test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) (owner: 10Ladsgroup) [17:02:52] (03PS2) 10Ladsgroup: First CI test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) [17:04:22] (03PS3) 10Ladsgroup: First CI test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) [18:47:59] Amir1, around? [18:48:13] yup [18:48:22] I'm looking for 4-5 sentences about fawikipedians getting sensitized to efficient vandalism detection. [18:48:32] My notes say "stadium" [18:49:00] yeah, the incident [18:49:56] someone changed architect of one of the stadiums for Euro soccer championship [18:50:22] to his name, the national reporter used that [18:50:32] than it went viral [18:50:51] people learned that they can change wikipedia, a huge wave of vandalism just started [18:51:05] at the exact time we deployed it in fawp [18:51:07] it helped [18:51:15] halfak: ^ [18:54:11] Perfect. thanks. [19:33:22] hi Amir1 [19:41:48] Amir1: I just read the weekly update...have you read the first sentence of https://ores.wikimedia.org/ ? ;) [19:42:30] lol darn it [19:42:40] * halfak will pick that up. [19:42:49] already working on it :P [19:42:57] BTW, I've been hacking on the outline here. [19:42:58] https://etherpad.wikimedia.org/p/ores_prod_blog [19:43:05] legoktm, that bit should be configurable. [19:43:18] ah, I was just gonna remove it [19:43:23] Actually now that I look at it, we should just remove it [19:43:27] +1 then :) [19:44:26] https://github.com/wiki-ai/ores/pull/160 [19:45:23] ... (ORES) is a web service running that provides machine learning ... [19:45:33] Probably want to remove "running" [19:45:45] legoktm, ^ [19:45:59] oops. [19:48:07] ERROR: Sorry, but @wiki-ai has blocked access to SSH keys created by some third-party applications. Your key was created before GitHub tracked keys created by applications, so we need your help. [19:48:39] halfak: updated [19:49:17] legoktm: halfak hey [19:49:24] I just got back from dinner [19:49:58] legoktm: {{merged}} [19:50:11] * halfak shakes fist at Amir1 for beating him to it. [19:50:20] :D [19:51:35] :) [19:51:40] * legoktm leaves for lunch [19:52:14] first commit to ores! [19:52:17] legoktm: for when you're back. I have so many patches in ORES extension waiting for review (and one in core too) [19:52:25] yay [20:53:30] halfak: https://etherpad.wikimedia.org/p/ores_prod_blog this looks great [21:06:25] 06Revision-Scoring-As-A-Service, 10ORES: ORES production blog post - https://phabricator.wikimedia.org/T141275#2497013 (10Halfak) [21:06:28] 06Revision-Scoring-As-A-Service, 10ORES: ORES production blog post - https://phabricator.wikimedia.org/T141275#2492654 (10Halfak) https://etherpad.wikimedia.org/p/ores_prod_blog [21:48:44] We should deploy wikilabels for the sake of UI translations [21:48:46] :D [21:51:24] Sure! Want to do that or should I? [21:51:39] Either way, please let me know so I can update the JS on-wiki [21:51:41] Amir1, ^ [21:52:01] sure. I'm about to finish this doc [21:52:07] halfak: after that I will do it [21:52:17] OK great [21:53:42] halfak: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/Deployment [21:53:45] this is very basic [21:54:01] lots of things are wrong. Just dumping everything in my head [21:54:43] It's a good start [21:54:50] Let me know when you are done and I'll take a pass [21:55:18] sure. I'm deploying right now [22:05:49] halfak: please update the js [22:06:11] * halfak waits for labels.wmflabs.org to load [22:06:23] 502 [22:06:29] There we go [22:07:03] {{done}} [22:07:03] You rule, halfak! [22:54:24] 06Revision-Scoring-As-A-Service, 10Wikilabels: Add wiki labels detail to deployment docs on wikitech - https://phabricator.wikimedia.org/T131768#2497261 (10Ladsgroup) a:03Ladsgroup [23:51:10] (03CR) 10Legoktm: [C: 032] Add revision_id to log for errors [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301136 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [23:52:28] (03CR) 10Legoktm: [C: 032] "This doesn't really test much, but is a good start :)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) (owner: 10Ladsgroup) [23:55:01] yayaya [23:56:35] (03Merged) 10jenkins-bot: Add revision_id to log for errors [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301136 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [23:56:38] (03Merged) 10jenkins-bot: First CI test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/301162 (https://phabricator.wikimedia.org/T140455) (owner: 10Ladsgroup)