[11:14:46] (03PS1) 10Ladsgroup: Performance boos on hidenondamaging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294693 (https://phabricator.wikimedia.org/T137895) [11:16:52] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2384952 (10akosiaris) Adding @mobrovac as during the Ops meeting it was suggested that reacting to RC events is something Change Propagation is designed to do. It might make sense to move t... [11:33:20] (03PS2) 10Ladsgroup: Performance boost on hidenondamaging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294693 (https://phabricator.wikimedia.org/T137895) [11:38:47] (03PS3) 10Ladsgroup: Performance boost on hidenondamaging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294693 (https://phabricator.wikimedia.org/T137895) [11:42:14] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2382984 (10Ladsgroup) a:03Ladsgroup [12:12:48] 06Revision-Scoring-As-A-Service, 10ORES, 06Operations: ORES should advertise swagger specs under /?spec - https://phabricator.wikimedia.org/T137804#2385011 (10Ladsgroup) [12:52:52] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385087 (10Ladsgroup) By using INNER JOIN instead of LEFT JOIN in case of hidenondamaging, everything would tons and tons of times faster. [12:55:58] 06Revision-Scoring-As-A-Service, 10ORES, 06Operations: ORES should advertise swagger specs under /?spec - https://phabricator.wikimedia.org/T137804#2385088 (10Ladsgroup) p:05Triage>03Unbreak! [13:04:19] (03CR) 10Daniel Kinzler: [C: 032] Performance boost on hidenondamaging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294693 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [13:04:36] o/ Amir1 [13:04:53] halfak: o/ [13:04:59] I have tons of news for you [13:05:28] (03Merged) 10jenkins-bot: Performance boost on hidenondamaging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294693 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [13:05:28] first we had down time for ores.wm.o [13:05:35] for eight hours [13:05:43] Eeek. What happened? [13:05:46] icingna didn't complain [13:05:49] Also I need to set up paging [13:05:50] Oh [13:05:50] got overloaded [13:06:06] I restarted it to clear the cache [13:06:07] The workers just gave up? [13:06:19] it seems so [13:06:24] there are two reasons behind it [13:06:42] 1- one of my friends ran stats on it, he is mad [13:06:52] (it's a long story) [13:07:09] he doesn't believe in throttling [13:07:29] (but we should have something to prevent such cases) [13:07:51] 2- we deployed the extension in wikidatawiki so basically we had two big precaching for wikidata. [13:08:05] I think you need to disable your precaching for wikidata [13:08:08] halfak: ^ [13:08:27] Why would we disable precaching for wikidata? [13:08:30] and obviously we need to fix icigna [13:08:48] Also, someone was running stats using ORES and that overloaded us? [13:08:49] because the extension does it right now [13:09:01] How many requests per seconds was he sending? [13:09:11] batches of 100 [13:09:50] he says he throttled for 5 secs but I don't believe him [13:10:10] he set all of throttles to zero in pywikibot [13:10:35] anyway, it might be an issue with other services, [13:10:49] mobrovc told us he disabled puppet in sc1001 [13:10:51] Amir1, no way that should have ever come close to hitting our capacity. [13:11:34] halfak: Actually I don't have an estimation of our capacity in prod [13:11:43] how big we can handle? [13:11:57] When I ran my test, I was running 2 requests for 50 in parallel. [13:11:59] 2 workers [13:12:09] No delay between requests. [13:17:37] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2385142 (10Halfak) @akosiaris, this sounds fine, but I don't see any links to "Change Propagation". Do you have suggestions on what steps we might go through to make use of it? In the en... [13:26:27] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2385148 (10akosiaris) >>! In T137370#2385142, @Halfak wrote: > @akosiaris, this sounds fine, but I don't see any links to "Change Propagation". Do you have suggestions on what steps we mig... [13:31:21] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Spec] Tracking and blocking specific IP/user-agent combinations - https://phabricator.wikimedia.org/T137962#2385178 (10Halfak) [13:39:44] (03PS1) 10Ladsgroup: Skip when an edit is errored in PopulateDatabase.php [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294711 (https://phabricator.wikimedia.org/T137854) [13:40:08] (03PS1) 10Ladsgroup: Performance boost on hidenondamaging [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294712 (https://phabricator.wikimedia.org/T137895) [13:47:38] (03CR) 10Jcrespo: "I've made some comments, but I am not a mediawiki developer, you should ask a more senior mediawiki hacker to review your patches." (032 comments) [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294712 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [13:50:05] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385248 (10jcrespo) Can you copy me the potential SQL queries generated by your code so I can run EXPLAIN on them on production to test them? Thank you. [13:52:20] (03CR) 10Ladsgroup: "This is not the main patch, this is the patch for backporting it to wmf.6. The main is https://gerrit.wikimedia.org/r/#/c/294693/ which go" (032 comments) [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294712 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [13:56:08] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385256 (10Ladsgroup) ``` EXPLAIN SELECT /* SpecialRecentChanges::doMainQuery Ladsgroup */ rc_id,rc_timestamp,rc_user,rc_user_text,rc_namespace,rc_ti... [14:14:37] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385293 (10jcrespo) I think the changes look good, 30K-100K rows visited using an index. It could be better,... [14:15:18] Amir1, can you re-hash your thoughts re. wikidata precaching in a task? [14:25:56] halfak: I thought since we do precaching in a better way in the extension (by omitting bots, etc.) [14:26:17] we should disable precaching for Wikidata (for now at least) [14:26:30] it would reduce the workload on the server greatly [14:26:50] Ahh. I see. We could also build this functionality into precached [14:27:05] E.g. right now we have "edit" as an event. We could have "nonbot_edit". [14:27:18] nice [14:29:48] but since the extension is doing it already for wikidata what's the point of running it? [14:31:30] Amir1, so that we have our needs met generally. [14:31:41] Also, goodfaith precaching [14:32:01] Which, BTW, I'd like to have as part of the ORES extension soon. [14:32:06] I'll file a task for that. [14:32:51] Amir1, what was the name you wanted to use for the UI? [14:35:24] halfak: I don't follow :D [14:35:43] https://phabricator.wikimedia.org/T137362 [14:36:57] ORES edit review or something like that [14:37:09] "ORES review tool" [14:37:20] https://www.mediawiki.org/wiki/ORES_review_tool [14:38:58] https://phabricator.wikimedia.org/T137966 [14:38:59] Thanks [14:40:52] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385421 (10Ladsgroup) @jcrespo I wrote there, I hope I understood you correctly, We can't get that threshold... [14:41:22] halfak: thanks :) [14:41:46] in twenty minutes, the performance issue for wikidata would be solved \o/ [14:42:10] Great. [14:42:55] 06Revision-Scoring-As-A-Service, 10ORES: Add 'nonbot_edit' event to precached - https://phabricator.wikimedia.org/T137967#2385423 (10Halfak) [14:42:59] https://phabricator.wikimedia.org/T137967 [14:48:16] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385445 (10jcrespo) No, I am talking about using a particular role when connecting to the database and selec... [14:51:45] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385459 (10Ladsgroup) Oh, this hook is not changing the DB slave role, It only adds condition and stuff, the... [15:09:41] (03CR) 10Thcipriani: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294711 (https://phabricator.wikimedia.org/T137854) (owner: 10Ladsgroup) [15:10:38] (03Merged) 10jenkins-bot: Skip when an edit is errored in PopulateDatabase.php [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294711 (https://phabricator.wikimedia.org/T137854) (owner: 10Ladsgroup) [15:22:59] (03CR) 10Thcipriani: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294712 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [15:24:01] (03Merged) 10jenkins-bot: Performance boost on hidenondamaging [extensions/ORES] (wmf/1.28.0-wmf.6) - 10https://gerrit.wikimedia.org/r/294712 (https://phabricator.wikimedia.org/T137895) (owner: 10Ladsgroup) [15:36:51] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Deploy ORES extension as a Beta feature - https://phabricator.wikimedia.org/T130210#2129780 (10Ladsgroup) a:03Ladsgroup [15:40:18] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 05WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385610 (10jcrespo) 05Open>03Resolved [16:02:30] 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2385677 (10Ladsgroup) https://github.com/wiki-ai/ores/pull/148 That would fix the most of it. We need some other things too [17:14:20] (03PS1) 10Ladsgroup: Add addQuotes to $threshold [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294740 [20:34:45] 06Revision-Scoring-As-A-Service, 10ORES: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2386716 (10Ladsgroup) That's an issue with scap::source since they can't use diffusion repos, https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/manifests/source.pp... [20:41:56] 06Revision-Scoring-As-A-Service, 10ORES: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2386783 (10Ladsgroup) >>! In T137819#2386716, @Ladsgroup wrote: > That's an issue with scap::source since they can't use diffusion repos, https://phabricator.wikimedia.org/diffusion/OPUP/browse... [21:15:31] * halfak installs goddamn scipy [22:47:49] (03CR) 10Legoktm: "I filed T138043 for skipping deleted edits properly." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294398 (https://phabricator.wikimedia.org/T137854) (owner: 10Ladsgroup)