[08:41:59] akosiaris: hey, good morning! Do you have some time to check this? https://gerrit.wikimedia.org/r/#/c/293904 [08:45:46] 06Revision-Scoring-As-A-Service, 10ORES: ORES /etc/ores does not get populated on worker nodes - https://phabricator.wikimedia.org/T137813#2393010 (10Ladsgroup) [08:46:14] 06Revision-Scoring-As-A-Service, 10ORES: ORES /etc/ores does not get populated on worker nodes - https://phabricator.wikimedia.org/T137813#2379936 (10Ladsgroup) https://gerrit.wikimedia.org/r/#/c/293904 [08:46:22] 06Revision-Scoring-As-A-Service, 10ORES: ORES /etc/ores does not get populated on worker nodes - https://phabricator.wikimedia.org/T137813#2393014 (10Ladsgroup) a:03Ladsgroup [10:16:09] (03CR) 10Thiemo Mättig (WMDE): [C: 031] Add addQuotes to $threshold (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294740 (owner: 10Ladsgroup) [11:12:11] (03CR) 10Daniel Kinzler: [C: 032] Add addQuotes to $threshold [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294740 (owner: 10Ladsgroup) [11:14:05] (03Merged) 10jenkins-bot: Add addQuotes to $threshold [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294740 (owner: 10Ladsgroup) [12:26:44] 06Revision-Scoring-As-A-Service, 10bwds, 05Wikimania-Hackathon-2016: Generate bad words for all languages more than 100K articles - https://phabricator.wikimedia.org/T134629#2393464 (10Psychoslave) @Ladsgroup if you intend Wikimania, and you have time for this, I wish we may talk about this task. [13:12:05] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2393543 (10Ladsgroup) I was deploying this but it seems we ran into the stupid race condition issue (which made ores goes down for twenty minutes again) so we rolled back... [13:26:31] o/ [13:26:33] Hey Amir1 [13:26:35] just got "in" [13:38:54] o/ schana [13:40:36] hey halfak [13:42:16] halfak: hey [13:42:22] I was afk to return the bike [13:42:33] I have some news for you if you haven't heard of it yet [13:42:53] schana, this is a heck of a code review. It seems that we're addressing all of your concerns about the norms of our projects in this one review. https://github.com/wiki-ai/ores/pull/147 [13:43:04] I'm not sure this is a healthy pattern. [13:43:14] "norms" --> "conventions" [13:44:10] are these conventions documented anywhere? [13:45:14] No, but you can see them if you look at the rest of the code. [13:45:24] As you might imagine, drawing up a style guide has been lower priority. [13:45:48] As I noted in one of my comments, main(), run(), etc. is common to all of the rest of the utilities. [13:45:52] As is docopty [13:48:48] Without knowing the purpose behind a particular convention, it can come across as perpetuating legacy code [13:49:08] schana, yes. Consistency is key IMO. [13:49:23] FWIW, I've been interpreting your questions and comments as "-1" on the PR. [13:49:27] Maybe I shouldn't? [13:49:36] yeah, that's definitely not how they're intended [13:49:49] OK. Can we merge working code then? [13:49:58] by all means [13:50:03] *working code with no strong objections. [14:02:38] Amir1, https://github.com/wiki-ai/ores/pull/147#discussion_r67694079 [14:02:53] I just saw it [14:03:03] halfak: should we discuss here and put results there [14:03:11] or do it there [14:03:25] Hmm... DIscuss here and results there. [14:04:30] first, what you are doing it seems to break stuff because sort only gets /etc/ores/foo *before* /srv/ [14:04:36] because e comes before s [14:04:47] Oh! [14:04:53] Yeah, not just sorting file names! [14:05:11] kk. Let me have a look at this. [14:05:14] so if it was possible to sort based on the last part, it would be more bearable [14:05:20] Can do that [14:05:23] Will be updated shortly. [14:05:57] I still think we should first sort and then join because /etc configs comes from puppet and /srv/ configs comes from the repo [14:06:14] and puppet configs in general should be able to override the repo configs [14:06:22] but I have no strong feelings about it [14:06:47] the biggest thing for me is last part sorting [14:07:43] Amir1, but /etc/ location and order is not hard-coded. [14:07:51] Would be best to fix sorting IMO [14:07:57] okay [14:08:01] :) your call [14:10:53] Almost ready. [14:11:01] Bah! I need to write the blog post draft too [14:11:02] Arg! [14:11:39] You know, I was thinking about schana's comment re. multiple commits in a PR with squashing. [14:11:52] It sounds like a pain since we don't necessarily want a single commit per PR [14:11:57] But I can see the value. [14:12:16] schana, were you imagining a single commit per PR? [14:12:16] *the squashing was optional [14:12:21] no [14:12:29] not if there are distinct things changing [14:12:43] (although, in that case, there should probably be multiple pull requests) [14:13:58] Not if they are dependent [14:14:10] Unless they are cross-repo, of course. [14:15:20] Amir1, please review [14:16:25] halfak: LGTM, have you tested it? [14:16:37] Yes. [14:17:07] I started up `application.wsgi` and `application.celery` to make sure they work. [14:17:25] At some point, we should have `application.wsgi` replace `dev_server` [14:17:39] halfak: I'm waiting for CI tests [14:17:55] but do you think we should write a blog post for ORES deployment? [14:18:05] shouldn't we do some more wikis first ;) [14:18:20] Amir1, just a draft [14:18:27] For Jeff to think about [14:18:27] kk [14:18:59] https://etherpad.wikimedia.org/p/ores_prod_announcement [14:33:28] Amir1, see https://etherpad.wikimedia.org/p/ores_prod_announcement [14:33:35] I have two outlines. What do you think? [14:34:20] checking [14:36:20] halfak: great! [14:41:58] * halfak works on revscoring etherpad in prep [14:42:46] 06Revision-Scoring-As-A-Service, 10bwds, 05Wikimania-Hackathon-2016: Generate bad words for all languages more than 100K articles - https://phabricator.wikimedia.org/T134629#2393740 (10Ladsgroup) >>! In T134629#2393464, @Psychoslave wrote: > @Ladsgroup if you intend Wikimania, and you have time for this, I w... [14:54:33] halfak, Hey [14:55:14] Yuvi replied to the Phab bug. https://phabricator.wikimedia.org/T138111 [14:55:27] Hey SoniWP [14:55:41] Just about to start a sequence of meetings that will take 3.5 hours :( [14:55:42] :((((( [14:55:55] Alright. I'm here for a while. [14:56:10] Ping me when you check out Yuvi's replies [14:56:22] And let me know how we are planning to progress with the queries [14:56:38] SoniWP, what is the status of those queries? [14:56:48] Have you had time to confirm that they are, in fact, getting killed? [14:56:58] I ran them again today. [14:57:12] Roughly 35 mins in, it was dead again [14:57:29] Sufficient reason to believe that the queries are getting killed, halfak [14:57:52] What error? [14:57:57] SoniWP: we had a down time today [14:58:12] around 11 AM UTC -ish [14:58:33] Welp. THats almost exactly when my queries were run [14:58:36] OK. Now running away. I'd take an the MySQL error and bring it to #wikimedia-labs along with the phab ticket [14:59:14] I'll retry the queries and hope to get an error out of them now [14:59:22] thanks [15:15:31] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 05Wikimania-Hackathon-2016: A training session on supporting ORES in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2393840 (10Halfak) [15:38:46] Hey folks. Please add any notes to the meeting agenda: https://etherpad.wikimedia.org/p/revscoring [15:59:58] Hey halfak and Amir1, I won't be able to make the meeting. Errands got in the way -_- [16:24:47] 06Revision-Scoring-As-A-Service, 10MediaWiki-Special-pages, 10MediaWiki-extensions-ORES, 05MW-1.28-release-notes, and 3 others: Templatize Special:Contributions lines and add proper hooks - https://phabricator.wikimedia.org/T122537#2394100 (10Halfak) [16:25:38] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2394102 (10Halfak) FYI: notes here https://etherpad.wikimedia.org/p/ores_prod_repos [16:28:37] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train a `reverted` model for jawiki - https://phabricator.wikimedia.org/T130869#2394111 (10Halfak) [16:30:09] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train a `reverted` model for jawiki - https://phabricator.wikimedia.org/T130869#2149067 (10Halfak) Moving this to the backlog. Still looking for a Japanese speaker to help us review our dataset for training/testing. @Johan, maybe you could help us... [16:30:58] 06Revision-Scoring-As-A-Service, 10ORES: Move static assets to wheels - https://phabricator.wikimedia.org/T137362#2394118 (10Halfak) [16:34:56] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2394123 (10Halfak) [16:35:22] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak) a:05Halfak>03Ladsgroup [16:37:15] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2394126 (10Halfak) a:05Halfak>03Ladsgroup [16:38:38] 06Revision-Scoring-As-A-Service, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2394132 (10Halfak) @Johan, have you had time to look at doing this landing page translation yet? [16:39:47] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2394137 (10Halfak) @schana Any thoughts on what direction to take this? [16:41:00] 06Revision-Scoring-As-A-Service: Develop training session material for Wikimania Hackathon - https://phabricator.wikimedia.org/T134785#2394139 (10Halfak) a:05Halfak>03Ladsgroup [16:43:56] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Spec] Tracking and blocking specific IP/user-agent combinations - https://phabricator.wikimedia.org/T137962#2385178 (10Halfak) @BBlack, we're looking at building good mechanisms for mitigating DOS attacks in #ORES. How is this done elsewhere? Any protips. [16:46:56] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: [Design] Namespace filter for ORES review tool highlighting - https://phabricator.wikimedia.org/T137844#2381039 (10Halfak) [16:47:48] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: [Design] Namespace filter for ORES review tool highlighting - https://phabricator.wikimedia.org/T137844#2381039 (10Halfak) @Yamaha5, it looks like Special:RecentChanges already is able to filter based on namespace. Can you just limit to usi... [16:51:40] I think I need lots of coffee this week :D [16:51:55] (I won't be the same person again after Wikimania :D) [16:54:36] halfak: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160620-ores [16:55:23] 06Revision-Scoring-As-A-Service, 10ORES: Document safe method to deploy ORES in prod - https://phabricator.wikimedia.org/T138234#2394194 (10Ladsgroup) [18:39:21] 06Revision-Scoring-As-A-Service, 10ORES: Document downtime for ORES - https://phabricator.wikimedia.org/T138248#2394697 (10Halfak) [18:39:25] 06Revision-Scoring-As-A-Service, 10ORES: Document downtime for ORES - https://phabricator.wikimedia.org/T138248#2394711 (10Halfak) https://wikitech.wikimedia.org/wiki/Incident_documentation/20160620-ores [18:39:32] * halfak makes phab card of downtime [18:41:06] * Amir1 thanks halfak [18:41:43] * halfak really wants to code, but will reluctantly work on task docs instead [18:44:37] 10Revision-Scoring-As-A-Service-Backlog, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2394727 (10Halfak) [18:45:24] 10Revision-Scoring-As-A-Service-Backlog, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2394740 (10Halfak) Hey @Capt_Swing, @DarTar suggested we reach out to #Design-Research and you specifically for a review of the ORES review... [18:47:55] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Discuss] Process for enabling ORES beta feature on big wikis - https://phabricator.wikimedia.org/T138250#2394743 (10Halfak) [18:48:36] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Discuss] Process for enabling ORES beta feature on big wikis - https://phabricator.wikimedia.org/T138250#2394756 (10Halfak) @Johan, could you comment on what concerns you see here? We've got the technical side down, but it would be nice to get your thoughts re... [18:50:07] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Halfak) [18:50:20] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: [Discuss] Process for enabling ORES beta feature on big wikis - https://phabricator.wikimedia.org/T138250#2394774 (10Halfak) [18:52:36] 06Revision-Scoring-As-A-Service, 10ORES: Document safe method to deploy ORES in prod - https://phabricator.wikimedia.org/T138234#2394777 (10Halfak) [18:52:38] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Documentation: Fix circular linking in ORES docs - https://phabricator.wikimedia.org/T138089#2394778 (10Halfak) [18:52:43] 06Revision-Scoring-As-A-Service, 10ORES: Add 'nonbot_edit' event to precached - https://phabricator.wikimedia.org/T137967#2394779 (10Halfak) [18:52:45] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2394780 (10Halfak) [18:52:47] 06Revision-Scoring-As-A-Service, 10ORES: Move static assets to wheels - https://phabricator.wikimedia.org/T137362#2394781 (10Halfak) [18:52:49] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394776 (10Halfak) [18:53:04] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: [Discuss] Process for enabling ORES beta feature on big wikis - https://phabricator.wikimedia.org/T138250#2394783 (10Halfak) [18:53:06] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Halfak) [18:53:41] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Halfak) [18:53:50] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Halfak) [18:56:38] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce ORES prod (technical lists) - https://phabricator.wikimedia.org/T138252#2394793 (10Halfak) [18:56:56] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce ORES prod (technical lists) - https://phabricator.wikimedia.org/T138252#2394808 (10Halfak) [18:58:44] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Convert ores-wikimedia-config to ores-wmflabs-deploy - https://phabricator.wikimedia.org/T138253#2394815 (10Halfak) [18:58:56] TASKS! [19:02:14] :D [19:52:42] Hey halfak [19:54:16] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Wikilabels UI reports non-200 status errors badly - https://phabricator.wikimedia.org/T138255#2394907 (10Halfak) [20:00:16] o/ [20:00:23] Sorry to miss your ping SoniWP [20:01:12] halfak, Check https://phabricator.wikimedia.org/T138111 [20:01:30] Something tells me the problem is not with the database, but the query we made [20:08:31] Hmm. yeah. This is not what I expected performance-wise. [20:08:34] * halfak checks. [20:08:51] Woah. looks like we are missing a group [20:11:17] Its probably got to do with rev_id and how the query was framed around it [20:11:51] I'm going to try some simpler queries to see if it works. Then try working through why the query is returning 0 elements [20:17:56] OK. I think I see a simplification [20:18:01] So we're really sampling from pages [20:18:33] I can scan 9m rows in the page table in 52 seconds [20:18:36] So that works [20:20:38] SoniWP, https://gist.github.com/halfak/f03bfea42d63824b1856fb60ae5e4aa6 [20:20:50] Replace "staging" with your user DB name [20:32:11] halfak, Gotcha. I need to leave now though. [20:32:24] So I'll only be able to tackle this tomorrow or whenever I get back next [20:32:36] OK. I think this one will run nicely when you are ready. [20:32:40] Have a good one! [20:32:41] o/ [20:33:37] ERROR 1044 (42000): Access denied for user 's53024'@'%' to database 'enwiki_p' [20:33:39] Woops [20:33:55] Any idea what this error is, halfak ? [20:34:06] Gotta make your own DB in order to make a temp table [20:34:25] u53024__sandbox_p [20:34:41] CREATE DATABASE u53024__sandbox_p [20:34:50] And then replace "staging" with that ugly name [20:35:39] halfak, Got it, thanks [20:35:47] I'll see you later. [20:35:48] \o [20:36:02] o/ [20:51:57] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce ORES prod (technical lists) - https://phabricator.wikimedia.org/T138252#2395113 (10Halfak) [20:52:00] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce ORES prod (technical lists) - https://phabricator.wikimedia.org/T138252#2394793 (10Halfak) https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service/Production_announcement [21:21:19] 10Revision-Scoring-As-A-Service-Backlog, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2395261 (10Capt_Swing) a:03Capt_Swing [21:31:53] 06Revision-Scoring-As-A-Service, 10ORES: Move static assets to wheels - https://phabricator.wikimedia.org/T137362#2395279 (10Halfak) Just set up https://github.com/wiki-ai/flask-wikimediaui. It seems to work nicely. [22:06:53] 06Revision-Scoring-As-A-Service, 10ORES: Move static assets to wheels - https://phabricator.wikimedia.org/T137362#2366069 (10Halfak) Just pushed the last changes to https://github.com/wiki-ai/ores/pull/149. It should be good to go. [22:07:45] OK. We now have a complete set of swagger and wikimedia-ui assets that can be wheel'd up :) [22:07:49] I'm out of here. [22:07:50] o/ [22:08:03] 06Revision-Scoring-As-A-Service, 10ORES: Add 'nonbot_edit' event to precached - https://phabricator.wikimedia.org/T137967#2395342 (10Halfak) a:03Halfak [22:08:29] 06Revision-Scoring-As-A-Service, 10ORES, 06Operations: ORES should advertise swagger specs under /?spec - https://phabricator.wikimedia.org/T137804#2395343 (10Halfak) a:03Halfak [22:08:41] 06Revision-Scoring-As-A-Service, 10ORES, 06Operations: ORES should advertise swagger specs under /?spec - https://phabricator.wikimedia.org/T137804#2379606 (10Halfak) p:05Unbreak!>03High [22:09:02] Claimed a few tasks for the morning that I think I can work through quickly [22:09:03] o/