[00:01:31] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Spike: [Spike] Should we make a model for ores in beta? - https://phabricator.wikimedia.org/T141980#2521305 (10Legoktm) >>! In T141980#2519511, @Halfak wrote: > However, I think it would be fine to ceas... [03:11:48] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:27] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 370 bytes in 5.795 second response time [03:27:18] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:16] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 385 bytes in 0.612 second response time [03:31:42] WHAT!? [03:31:44] Arg [03:34:29] OK. I think that -05 might get rebuilt, but it's waiting until the AM UTC-5 [03:34:37] I'll be back if it happens again. [06:08:49] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-Incident: Set up password on ORES Beta redis server - https://phabricator.wikimedia.org/T141823#2521531 (10Ladsgroup) a:03Ladsgroup [06:08:54] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-Incident: Set up password on ORES Beta redis server - https://phabricator.wikimedia.org/T141823#2513386 (10Ladsgroup) https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=816271&oldid=739698 (And some restarts here and there) [06:48:00] schana/revscoring#1 (tamil - c2b7344 : Nathaniel Schaaf): The build failed. https://travis-ci.org/schana/revscoring/builds/149669446 [06:50:47] schana/revscoring#2 (tamil - e9e2d9a : Nathaniel Schaaf): The build failed. https://travis-ci.org/schana/revscoring/builds/149669873 [06:59:18] 06Revision-Scoring-As-A-Service, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2521576 (10schana) It seems like the `aspell-ta` package [[https://travis-ci.org/schana/revscoring/builds/149669873|isn't working on travis]] (even though it installed correctly). Without alterna... [13:45:09] (03CR) 10Daniel Kinzler: "Intent looks good. I don't know the services used here, so i can't vouch for functionality." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/302703 (https://phabricator.wikimedia.org/T141978) (owner: 10Ladsgroup) [14:02:31] o/ Amir1 [14:06:10] 06Revision-Scoring-As-A-Service, 10ORES: Extrapolate memory usage per worker forward 2 years - https://phabricator.wikimedia.org/T142046#2522222 (10Halfak) p:05Triage>03High [14:07:23] 10Revision-Scoring-As-A-Service-Backlog: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2522225 (10Halfak) I looked into this with @akosiaris. We think this issue was cased by the redis node going into swap. We started up `ores-redis-02` with more memory and... [14:08:00] 06Revision-Scoring-As-A-Service, 10ORES: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2522227 (10Halfak) [14:08:16] 06Revision-Scoring-As-A-Service, 10ORES: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2517410 (10Halfak) a:03Halfak [14:11:59] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Review changes to graphite/grafana for ORES refactor - https://phabricator.wikimedia.org/T141781#2522235 (10Halfak) Note #1: https://grafana.wikimedia.org/dashboard/db/ores-labs is dead and we should move this board to https://grafana-labs.wikimedia.org/. Tha... [14:12:36] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Review changes to graphite/grafana for ORES refactor - https://phabricator.wikimedia.org/T141781#2522237 (10Halfak) p:05Triage>03Normal [14:13:58] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 10rsaas-editquality: Write proof-of-concept good-faith newcomer detection - https://phabricator.wikimedia.org/T141779#2522242 (10Halfak) p:05Triage>03Normal [14:14:50] 10Revision-Scoring-As-A-Service-Backlog, 06Research-and-Data-Backlog, 10Wikilabels, 10rsaas-edittypes, 03Research-and-Data-2016-Q3: Train edit types model on labeled data for English Wikipedia - https://phabricator.wikimedia.org/T121715#2522244 (10Halfak) p:05Triage>03Lowest [14:15:10] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Extend the GUI for ORES - https://phabricator.wikimedia.org/T107723#2522245 (10Halfak) p:05Triage>03Low [14:15:48] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for dewiki - https://phabricator.wikimedia.org/T130257#2522247 (10Halfak) p:05Triage>03Lowest [14:15:54] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for viwiki - https://phabricator.wikimedia.org/T130300#2522248 (10Halfak) p:05Triage>03Lowest [14:16:08] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130296#2522252 (10Halfak) p:05Triage>03Lowest [14:17:15] 06Revision-Scoring-As-A-Service, 10ORES: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2522253 (10akosiaris) >>! In T141946#2522225, @Halfak wrote: > I looked into this with @akosiaris. We think this issue was cased by the redis node going into swap. Som... [14:17:17] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete ukwiki edit quality campaign - https://phabricator.wikimedia.org/T130271#2522255 (10Halfak) [14:17:19] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130296#2131874 (10Halfak) [14:17:21] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete ukwiki edit quality campaign - https://phabricator.wikimedia.org/T130271#2131414 (10Halfak) [14:17:23] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130296#2131874 (10Halfak) [14:18:07] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete ukwiki edit quality campaign - https://phabricator.wikimedia.org/T130271#2131414 (10Halfak) [14:18:09] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130294#2522263 (10Halfak) [14:18:39] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130294#2131841 (10Halfak) [14:18:41] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete ukwiki edit quality campaign - https://phabricator.wikimedia.org/T130271#2131414 (10Halfak) [14:18:43] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ukwiki - https://phabricator.wikimedia.org/T130296#2522264 (10Halfak) [14:19:44] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Deploy edit quality models for plwiki - https://phabricator.wikimedia.org/T130292#2522267 (10Halfak) [14:21:07] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Deploy edit quality models for plwiki - https://phabricator.wikimedia.org/T130292#2131806 (10Halfak) Models have been trained and are waiting for a deploy. This almost happened already. See https://wikitech.wikimedia.org/wiki/Incident_documentation/201608... [14:22:48] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for eswiki - https://phabricator.wikimedia.org/T130279#2522275 (10Halfak) p:05Triage>03Lowest [14:22:50] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for azwiki - https://phabricator.wikimedia.org/T130278#2522276 (10Halfak) p:05Triage>03Lowest [14:22:54] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for frwiki - https://phabricator.wikimedia.org/T130282#2522277 (10Halfak) p:05Triage>03Lowest [14:23:00] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for etwiki - https://phabricator.wikimedia.org/T130280#2522279 (10Halfak) p:05Triage>03Lowest [14:23:07] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#2522280 (10Halfak) p:05Triage>03Lowest [14:23:14] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for idwiki - https://phabricator.wikimedia.org/T130285#2522281 (10Halfak) p:05Triage>03Lowest [14:23:20] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for itwiki - https://phabricator.wikimedia.org/T130286#2522282 (10Halfak) p:05Triage>03Lowest [14:23:29] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for nowiki - https://phabricator.wikimedia.org/T130291#2522283 (10Halfak) p:05Triage>03Lowest [14:23:36] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for jawiki - https://phabricator.wikimedia.org/T130288#2522286 (10Halfak) p:05Triage>03Lowest [14:24:56] 10Revision-Scoring-As-A-Service-Backlog, 07Spike: [Spike] Could we build a "list article" quality model - https://phabricator.wikimedia.org/T136006#2522288 (10Halfak) p:05Triage>03Lowest [14:27:19] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Request raw features/datasources from ORES - https://phabricator.wikimedia.org/T133585#2522296 (10Halfak) [14:28:21] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Request raw features/datasources from ORES - https://phabricator.wikimedia.org/T133585#2236594 (10Halfak) p:05Triage>03Lowest [14:28:45] 10Revision-Scoring-As-A-Service-Backlog: [Discuss] Opportunities for crowdsourcing around ORES - https://phabricator.wikimedia.org/T132440#2522300 (10Halfak) 05Open>03Invalid [14:29:11] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-articlequality : NLP for article quality models. - https://phabricator.wikimedia.org/T132533#2522302 (10Halfak) p:05Triage>03Low [14:30:58] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-FlaggedRevs: Use revision scoring to trigger flagged protection - https://phabricator.wikimedia.org/T132901#2522311 (10Halfak) p:05Triage>03Low [14:31:21] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 07Spike: [Spike] investigate using compression ratios in revscoring - https://phabricator.wikimedia.org/T132375#2522313 (10Halfak) p:05Triage>03Lowest [14:32:40] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Train/test copyvio detection model - https://phabricator.wikimedia.org/T131481#2522329 (10Halfak) p:05Triage>03Normal [14:35:09] 06Revision-Scoring-As-A-Service, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Finding items that should be merged - https://phabricator.wikimedia.org/T127467#2522332 (10Halfak) [14:36:10] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 10rsaas-editquality: Write proof-of-concept good-faith newcomer detection - https://phabricator.wikimedia.org/T141779#2522336 (10Halfak) [14:36:12] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Finding good-faith contributors with ORES - https://phabricator.wikimedia.org/T127474#2522338 (10Halfak) [14:36:30] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 10rsaas-editquality: Write proof-of-concept good-faith newcomer detection - https://phabricator.wikimedia.org/T141779#2511851 (10Halfak) [14:37:15] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 10rsaas-editquality: Write proof-of-concept good-faith newcomer detection - https://phabricator.wikimedia.org/T141779#2511851 (10Halfak) [14:38:28] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: More robust/automated deployment of ORES - https://phabricator.wikimedia.org/T130410#2522362 (10Halfak) This has been implemented with scap3. We now do deployments within the appropriate windows. [14:38:33] 06Revision-Scoring-As-A-Service, 10ORES, 07Epic: [Epic] Structured deployment of ORES - https://phabricator.wikimedia.org/T130369#2522364 (10Halfak) [14:38:34] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: More robust/automated deployment of ORES - https://phabricator.wikimedia.org/T130410#2522363 (10Halfak) 05Open>03Resolved [14:39:15] 10Revision-Scoring-As-A-Service-Backlog: Clustering of Bad edits - https://phabricator.wikimedia.org/T113919#1679798 (10Halfak) We looked into this and it didn't really make that much sense. [14:39:28] 10Revision-Scoring-As-A-Service-Backlog: Clustering of Bad edits - https://phabricator.wikimedia.org/T113919#2522369 (10Halfak) 05Open>03declined [14:40:04] 10Revision-Scoring-As-A-Service-Backlog: Look into matching images of the same painting - https://phabricator.wikimedia.org/T131553#2522372 (10Halfak) We'd need someone to look into image processing strategies in order to get started on this. [14:40:12] 10Revision-Scoring-As-A-Service-Backlog: Look into matching images of the same painting - https://phabricator.wikimedia.org/T131553#2522373 (10Halfak) p:05Triage>03Lowest [14:41:17] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Increase signal of feature set for Wikidata model - https://phabricator.wikimedia.org/T127473#2522375 (10Halfak) [14:41:24] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Increase signal of feature set for Wikidata model - https://phabricator.wikimedia.org/T127473#2044563 (10Halfak) p:05Triage>03Normal [14:46:03] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Wikidata item quality classification - https://phabricator.wikimedia.org/T127470#2522409 (10Halfak) p:05High>03Normal [14:47:21] 10Revision-Scoring-As-A-Service-Backlog, 07Spike: [Spike] Explore how we could train models on suppressed and revdeleted content - https://phabricator.wikimedia.org/T102344#2522424 (10Halfak) We can now log in when processing feature extraction. We'll need to be able to log in for revert detection. We'll som... [14:48:19] 10Revision-Scoring-As-A-Service-Backlog, 07Spike: [Spike] Explore how we could train models on suppressed and revdeleted content - https://phabricator.wikimedia.org/T102344#2522431 (10Halfak) [14:48:22] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Train/test copyvio detection model - https://phabricator.wikimedia.org/T131481#2522430 (10Halfak) [14:49:05] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 07Spike: [Spike] Investigate stemmer for Turkish: Zemberek - https://phabricator.wikimedia.org/T102337#2522438 (10Halfak) p:05Normal>03Lowest [14:49:29] 10Revision-Scoring-As-A-Service-Backlog, 07Documentation: Document interesting potential uses of revscores - https://phabricator.wikimedia.org/T102339#2522442 (10Halfak) 05Open>03Resolved [14:50:48] 10Revision-Scoring-As-A-Service-Backlog, 07Upstream: Add turkish stemmer to NLTK - https://phabricator.wikimedia.org/T102340#2522446 (10Halfak) p:05Triage>03Lowest [14:57:23] * halfak laughs at Amir1's "areallysecretpassword" [14:57:26] :D [15:01:47] schana/revscoring#3 (tamil - 26cbfb1 : Nathaniel Schaaf): The build was fixed. https://travis-ci.org/schana/revscoring/builds/149782653 [15:07:48] 06Revision-Scoring-As-A-Service, 10ORES: Planning long-term ORES architecture - https://phabricator.wikimedia.org/T137125#2522494 (10schana) 05Open>03Invalid [16:46:48] halfak: hey, tomorrow is the celebration of 500K articles of Persian Wikipedia. We made site notice and this is going to be big (the first dedicated conference for Wikipedia in Iran) I'm a speaker and organizer [16:47:01] I should've told you about it sooner. I'm sorry [16:47:20] but I added redis password to the beta instance today [16:47:22] No worries. Have fun! [16:47:28] I saw that. Thanks for getting to it [16:47:35] I had a good laugh at the password :) [16:47:38] ^_^ [16:47:42] :D [16:47:46] I'm just finishing up the worker memory estimates. [17:37:38] Check it out: https://commons.wikimedia.org/wiki/File:Ores_worker_memory.linear_model_extrapolation.svg [17:38:02] Looks like our workers will probably need ~2.15 GB of memory in 2 years if we keep up the pace. [17:38:13] akosiaris, ^ if you're still around. [17:38:21] I'll update the phab ticket with this plot. [17:44:32] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:40] No [17:45:09] Weird. I can't ssh to web-05 [17:45:22] ssh_exchange_identification: Connection closed by remote host [17:46:02] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:32] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 385 bytes in 4.678 second response time [17:46:44] WTF was that? [17:47:02] ores-web-05 now wants a password for logging in [17:47:53] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 369 bytes in 0.698 second response time [17:48:48] halfak: remember we need to make puppet changes too. [17:49:32] We should write them as a backward-compatible way which is possible but not a little bit complicated [17:50:01] "not" was redundant [17:51:09] (I wrote all of that in the incident report talk page, and I guess no one read those) [17:51:16] :D [17:51:21] not a single feedback [17:53:52] Oh, we had a feedback, why didn't I get a ping. https://wikitech.wikimedia.org/wiki/Talk:Incident_documentation/20160801-ORES [17:53:54] anyway [18:03:25] Yeah. I hear you. When is our next window to deploy? [18:03:34] Amir1, should we schedule a deploy party? [18:05:47] 06Revision-Scoring-As-A-Service, 10ORES: Extrapolate memory usage per worker forward 2 years - https://phabricator.wikimedia.org/T142046#2523233 (10Halfak) OK. Extrapolation complete. See https://commons.wikimedia.org/wiki/File:Ores_worker_memory.linear_model_extrapolation.svg for the plot. If we keep up... [18:06:06] 06Revision-Scoring-As-A-Service, 10ORES: Extrapolate memory usage per worker forward 2 years - https://phabricator.wikimedia.org/T142046#2523234 (10Halfak) @akosiaris, is ^ crazy? [18:07:13] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 324 bytes in 0.132 second response time [18:12:07] Well... ores-web-05 is officially totally insane [18:12:11] I can't even run puppet [18:17:13] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 370 bytes in 8.901 second response time [18:22:44] halfak: I was afk [18:23:03] I just put out a big fire with andrewboggot's help. [18:23:06] halfak: https://wikitech.wikimedia.org/wiki/Deployments [18:23:11] ores-web-05 is a pile of poop [18:23:27] But I'm just about to turn ores-web-04 back on since I think I know what was causing the issues. [18:23:44] Also, I'm moving the labs dashboard. [18:23:48] I'm almost done [18:23:58] Monday "20:00–21:00 UTC" [18:24:04] Monday looks good. [18:24:07] Would that time work for you? [18:24:28] Looks like it's a little late [18:24:39] it's fine [18:24:46] We could make sure to have the ducks in a row earlier in the day and you could leave the deploy to me [18:24:52] I work that late usually [18:25:04] no, it's okay [18:25:04] :) OK [18:25:21] Amir1, check this out: https://commons.wikimedia.org/wiki/File:Ores_worker_memory.linear_model_extrapolation.svg [18:25:46] once you're done with labs, did you check why beta acts strangely towards the "scores" key? [18:25:53] yeah. Will do [18:26:00] yup, I saw it, fantastic [18:26:10] Arg. I have 30 minutes before I need to drive somewhere. [18:26:14] * halfak goes extra fast [18:29:02] Amir1, if you want to look at https://grafana-labs-admin.wikimedia.org/dashboard/db/ores-labs, I'll move on to beta stuff. [18:29:15] I've got it mostly working [18:30:17] sure [18:30:53] we should have something for failed jobs too [18:30:58] we will work on it [18:34:00] I'm working on ores labs dashboard [18:34:44] OK. I think that the issue you found might have been due to some stuff that got added to the cache inappropriately. [18:37:11] I closed this tab by mistake :S [18:38:14] :D [18:38:21] I pinned this tab [18:38:36] Weird. It looks like we're running the right code in beta. But I can't replicate the issue locally. [18:38:56] I'm going to bed [18:39:14] I will talk to you later [18:39:16] o/ [18:39:18] OK. have a good one! [18:39:19] o/ [18:40:13] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES: Dashboard or pane for ORES failed jobs on beta - https://phabricator.wikimedia.org/T142119#2523395 (10Ladsgroup) [19:00:46] OK the next thing I want to do re. beta is clear the cache. [19:00:57] I can't seem to connect to the redis server on port 6380. [19:01:02] Only the one on 6379 [19:01:07] So I can't clear the cache. [19:01:19] I'm about to change locations and get lunch, so I'll be back in 1 hour. [20:43:39] Forgot to say "back" [20:43:47] But I'm here again now [20:43:51] Hacking on ORES documentation. [20:44:02] Once I'm done, i'll be looking into the beta cache again [21:48:00] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Make user-centered documentation for review tool - https://phabricator.wikimedia.org/T140150#2524227 (10Halfak) https://www.mediawiki.org/w/index.php?title=ORES_review_tool&type=revision&diff=2208338&oldid=2196685 BAM [21:48:22] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Make user-centered documentation for review tool - https://phabricator.wikimedia.org/T140150#2524228 (10Halfak) [21:48:36] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Make user-centered documentation for review tool - https://phabricator.wikimedia.org/T140150#2454899 (10Halfak) a:03Halfak [21:49:32] 06Revision-Scoring-As-A-Service, 10ORES: Investigate web-05 downtime - https://phabricator.wikimedia.org/T141523#2501811 (10Halfak) It looks like there's some OOM issues on web-05, so I reduced the amount of workers and it seems that we're doing better. I'm calling this done.