[00:32:22] o/ [00:32:33] (03PS1) 10Awight: [WIP] Validate global user ID in endorsements [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 [00:39:19] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:32] 10Scoring-platform-team, 10JADE, 10Documentation: Write glossary of JADE concepts - https://phabricator.wikimedia.org/T204905 (10Zppix) @awight this may be a good task for GCI I'd be willing to mentor or if you want you could mentor... [02:09:23] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:27] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:20] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:58:19] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 890 bytes in 0.303 second response time [05:08:00] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:00] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 890 bytes in 0.313 second response time [05:09:31] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:35] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:08] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10jcrespo) That looks really bad performance. Not only that scans the revision table from top to bottom (>200GB of data)... [07:30:14] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10Bawolff) >>! In T202596#4600586, @jcrespo wrote: > That looks really bad performance. Not only that scans the revision... [07:42:42] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10jcrespo) @Bawolff That new query you propose makes no sense to me- it just selects the first 100 revisions every singl... [08:09:38] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:34] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461568 (owner: 10L10n-bot) [09:29:33] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461572 (owner: 10L10n-bot) [09:39:42] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:18] Morgen o/ [10:35:26] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: ORES service gets stuck reporting "server overloaded" even after load returns to normal - https://phabricator.wikimedia.org/T175654 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup This is done. [10:50:01] I need to check the flip flopping ^ [11:09:47] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:57] afk for haircut. will be back soon [11:17:02] it's way overdue :D [12:39:51] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:19] back now [14:09:55] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:50] do we have experimental redis node? why? [14:25:39] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Implement whitelisting for IPs for ORES - https://phabricator.wikimedia.org/T204955 (10Ladsgroup) [14:28:00] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: ORES should return 429 instead of 408 in case of too many requests - https://phabricator.wikimedia.org/T204956 (10Ladsgroup) [14:53:45] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10hashar) [14:54:06] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10hashar) [14:56:01] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10hashar) Though if I try a reported URL manually, it seems to work. Tried on deploy1001, deploy2001 and mw2219 [15:03:54] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10hashar) Based on Graphana, seems some ORES server got overloaded and some requests did time out. That was a short break though and apparently everyth... [15:13:31] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10Ladsgroup) Yes, basically for each deployment we get an overload error spike: https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=9&fu... [15:19:45] 10Scoring-platform-team, 10ORES: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10Ladsgroup) I don't think it should be a blocker to the train as it's due to changes happening to the service ({T160692}) but it should be pretty high... [15:21:05] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review, 10User-Ladsgroup: ORES should return 429 instead of 408 in case of too many requests - https://phabricator.wikimedia.org/T204956 (10Ladsgroup) https://github.com/wikimedia/ores/pull/263/files [15:25:58] wikimedia/ores#1000 (error_429 - a0ca549 : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/ores/builds/431079574 [15:27:25] awight: for when you're around: https://github.com/wikimedia/ores/pull/263/files [15:27:32] I'm going for lunch [15:27:38] will be back soon [15:40:00] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:47] back now [16:38:01] One more crazy thing I need to deal with this morning, I'll be ready to work ~17:30 UTC [16:38:54] ^ merged :D [16:40:07] Thank you! [16:44:05] 10Scoring-platform-team (Current), 10ORES, 10WMF-JobQueue, 10Services (watching), 10Wikimedia-production-error: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T204753 (10Ladsgroup) >>! In T204753#4599481, @Krinkle wrote: > Should the following error have have its own task, or... [16:47:39] (03PS1) 10Ladsgroup: Bump ORES to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/461687 (https://phabricator.wikimedia.org/T204956) [16:49:31] (03CR) 10Ladsgroup: [V: 032 C: 032] Bump ORES to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/461687 (https://phabricator.wikimedia.org/T204956) (owner: 10Ladsgroup) [17:10:04] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:08] Looks like I get internal server errors if I ask for articlequality on multiple revisions where one is out of the normal range. https://ores.wikimedia.org/v3/scores/enwiki/?models=articlequality&revids=64196208800|123456&features [17:29:56] ragesoss: Nice bug report. That won't happen in practice though, I hope? [17:30:15] Also noticing that I get a lot of too_many_requests errors locally, even just testing the API manually in the browser. Like, I'm getting dinged for requests that already finished? [17:30:33] Amir1: ^ That would be related to ongoing work with PoolCounter [17:30:49] That shouldn't happen [17:30:57] on it [17:31:29] awight: no, that won't happen in practice. just putting the multiple-revs-per-request refactor through its paces... [17:32:19] I was mainly worried about whether there were URI length limits that would stop me from getting data for 50 revs at once, so I gradually lengthened the size of the revids until I got that error. [17:32:27] That's good information--generally we try to return all the good results and if a few revisions error (e.g., because they were deleted), we return an error in a container under that revid. [17:32:32] aah [17:32:46] URIs are infinite in theory, AFAIK [17:33:09] yeah, but lots of libraries subscribe to a different reality. [17:33:09] the limit is 2k chars for browsers IIRC [17:33:12] usually there's an upper bound imposed by browsers, servers ets [17:33:54] so, 50 rev ids should be no problem. [17:34:03] the error here is not about pool counter, it can't connect to enwiki [17:34:03] at least with a 2k limit. [17:34:10] which I assumed, but wanted to double check. [17:34:23] that's pretty weird [17:35:00] 10 chars for 9-digit revision plus "|" x 50 revs would be 500 chars, so we shouldn't hit any limits [17:36:47] response length may hit the limit like I was saying yesterday [17:37:50] ?features= responses seem to weigh in at around 12k for the moment. [17:38:12] 50 x 12k = 600k [17:38:41] IIRC, the response limit is around 1.5MB, though I'm not sure where that's configured, just vaguely remember from seeing it quoted in error logs. [17:42:24] app.log.2018-09-14:2018-09-16 02:22:25,968 WARNING mwapi.session: - result -- {'*': 'This result was truncated because it would otherwise be larger than the limit of 12,582,912 bytes.'} [17:42:30] 12.5M, we're good. [17:43:06] (exactly 12MB) [17:50:06] 10Scoring-platform-team, 10ORES: Read timeout from enwiki when requesting non-existent revision - https://phabricator.wikimedia.org/T204984 (10awight) [18:00:31] 10Scoring-platform-team, 10ORES: Read timeout from enwiki when requesting non-existent revision - https://phabricator.wikimedia.org/T204984 (10Ladsgroup) This is pretty funny because for one revision it works as expected: https://ores.wikimedia.org/v3/scores/enwiki/?models=articlequality&revids=64196208800&fea... [18:02:41] I need to relocate, will be back soon [18:40:07] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:47] Amir1: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/460995/ [18:47:01] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/461255/ [18:53:54] back now [18:53:58] will check [18:54:09] o/ [19:15:11] (03PS1) 10Ladsgroup: Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) [19:16:28] (03CR) 10jerkins-bot: [V: 04-1] Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:19:50] (03CR) 10jerkins-bot: [V: 04-1] Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:20:10] awight: https://github.com/wikimedia/ores/pull/264/files [19:22:18] (03PS2) 10Ladsgroup: Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) [19:22:53] wikimedia/ores#1003 (right_http_code - 0c8ebeb : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/ores/builds/431182445 [19:25:38] (03CR) 10Awight: Retry once when ores service is timed out (033 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:26:12] (03CR) 10Awight: [C: 032] Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:26:38] Awesome! [19:26:48] Thank you awight [19:28:06] (03CR) 10Ladsgroup: [C: 032] Rename namespace to NS_JUDGMENT [extensions/JADE] - 10https://gerrit.wikimedia.org/r/460995 (owner: 10Awight) [19:30:49] np! [19:30:56] likewise [19:31:37] Our logstash logs are a total mess [19:31:46] nothing good I can get out of it [19:32:44] These darn things are still outstanding, eh: T169586, T168921 [19:32:45] T168921: Send error logs to logstash - https://phabricator.wikimedia.org/T168921 [19:32:45] T169586: Send celery logs and events to logstash - https://phabricator.wikimedia.org/T169586 [19:32:58] I usually log into ores*.wmnet [19:32:59] (03Abandoned) 10Sbisson: Drop all trace of the Beta Features integration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/354729 (owner: 10Jforrester) [19:33:03] /srv/log/ores/app.log [19:33:27] I'd like to read main.log too, but the permissions are wrong and SRE thinking we should just wire to logstash rather than muck around in puppet. [19:34:08] The thing is they are not structured so it's super hard to find error in logstash [19:34:18] celery ones are not there at all [19:34:29] I think I will pick this up next [19:34:34] <3 thank you! [19:34:55] I'm procrastinating from writing a feature tied into CentralAuth. [19:35:04] Not sure how to mock the tests, etc. [19:35:36] (03Merged) 10jenkins-bot: Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:36:02] Have fun! I tried and failed several times [19:36:05] just to install it [19:38:18] 10Scoring-platform-team (Current), 10ORES, 10WMF-JobQueue, 10Patch-For-Review, and 2 others: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T204753 (10Krinkle) >>! In T204753#4602698, @Ladsgroup wrote: > It's not much and it's expected but it should just retry instead of errori... [19:40:07] (03CR) 10jenkins-bot: Retry once when ores service is timed out [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461710 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup) [19:43:21] (03Merged) 10jenkins-bot: Rename namespace to NS_JUDGMENT [extensions/JADE] - 10https://gerrit.wikimedia.org/r/460995 (owner: 10Awight) [19:44:11] yesss I was hoping it would be that bad [19:49:17] Amir1: FYI I have proposed a config patch for wp10/articlequality. We could potentially SWAT it on Monday, then merge the related PageTriage patch and let it ride the train. If all goes well we could cleanup the references to wp10 the week after. [19:52:16] stephanebisson: Thanks for helping us with this sanity-protecting measure :-) [19:53:30] stephanebisson: Thanks, I wrote a note there [19:53:57] yeah, I can split it [19:54:35] (03CR) 10jenkins-bot: Rename namespace to NS_JUDGMENT [extensions/JADE] - 10https://gerrit.wikimedia.org/r/460995 (owner: 10Awight) [19:57:30] Thank you! [19:58:55] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Implement whitelisting for IPs for ORES - https://phabricator.wikimedia.org/T204955 (10Ladsgroup) https://github.com/wikimedia/ores/pull/265 [20:00:23] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: ORES requests for wikidatawiki models=damaging end up with HTTP request timed out - https://phabricator.wikimedia.org/T204961 (10Ladsgroup) a:03Ladsgroup [20:00:47] wikimedia/ores#1006 (whitelisting - 862e3ec : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/ores/builds/431198012 [20:01:31] (03PS1) 10Awight: Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 [20:01:33] (03PS1) 10Awight: Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 [20:01:44] Amir1: done. I'll ping you on Monday about the plan above. [20:02:12] Sure, have a nice weekend! [20:02:19] Thanks, you too! [20:03:05] (03CR) 10jerkins-bot: [V: 04-1] Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 (owner: 10Awight) [20:03:14] (03CR) 10jerkins-bot: [V: 04-1] Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 (owner: 10Awight) [20:07:03] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Ladsgroup) @Elisardojm Thank you! It's great. One thing left for me before moving on is list of informal words that you can extrac... [20:09:41] (03CR) 10jerkins-bot: [V: 04-1] Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 (owner: 10Awight) [20:10:11] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:19] (03CR) 10jerkins-bot: [V: 04-1] Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 (owner: 10Awight) [20:20:18] bussing for 1hr [20:56:49] I'm guessing you're already well aware, but I'm getting a lot of the internal server errors with timeouts for en.wikipedia.org on requests for valid revisions as well. [20:58:45] err... nevermind... [20:59:02] my tests are failing because i left that bad revision in the test. [21:38:46] ragesoss: cheap thrills :-) [21:38:58] I wonder what happens when I ask MW for that revision directly... [21:39:28] when I only asked for the bad revision by itself, I got a normal error message that it doesn't exist. [21:40:15] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:34] no big deal for the MW backend, either: https://en.wikipedia.org/w/api.php?action=query&prop=revisions&revids=64196208800&rvslots=*&rvprop=content [21:43:39] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Elisardojm) @Ladsgroup Done, but I have one question. Do I have to add words with and without accents? [21:43:39] nice corner case! [21:52:19] harej: Found a weird edge case that I'm not going to worry about. Let's say there are two judgments for a diff, {damaging: true} and {damaging: true, goodfaith: false} [21:52:51] They can both be !preferred and it seems like we could assume that {damaging: true, goodfaith: false} is the consensus. [21:52:57] But for now, I'm going to call that a validation error. [21:53:19] Because of the lack of preferred flag? [21:53:33] Also, they're technically two different statements, so it would make sense to treat them differently. [21:53:40] Treat them as different objects, I mean. [21:54:09] The difference between the two is that the former is "there is no decision for goodfaith" which is basically a declaration of the absence of an opinion. a sort of "no contest" [21:54:39] So I would agree in not writing special rules around this case. [21:56:27] (03PS2) 10Awight: Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 [21:56:29] (03PS2) 10Awight: Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 [21:57:31] yeah exactly, here check this out: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/JADE/+/4a3e6024c0cf5e3ec647b2bf0bb521ec53dd8e75/includes/JudgmentValidator.php#106 [21:57:44] It should reflect what we talked about the other day. [21:58:20] PHP7 null coalesce FTW: $judgment->preferred ?? false [21:58:32] property coalescing might be... undocumented. but who's stopping us... [22:00:27] so, a thought. whether or not the preferred flag is regarded is left up to client code, yeah? [22:00:28] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Ladsgroup) >>! In T201142#4604150, @Elisardojm wrote: > @Ladsgroup Done Thank you! > I have one question. Do I have to add words w... [22:00:58] I would consider a page with 2 judgments and a page with 1 judgment to be meaningfully different, even if both have the same statement as the preferred one [22:02:22] harej: Yes, the client will receive all judgments on the page. [22:02:27] s/the/a/ [22:05:05] good work with the preferred validation, by the way [22:05:55] :) it reads well, I think. [22:06:36] One thing that I need to fix sooner rather than later: T197653 [22:06:37] T197653: Surface JADE validation errors - https://phabricator.wikimedia.org/T197653 [22:06:46] That takes some upstream work, I think, in mw-core. [22:08:08] What does JsonContent do? [22:09:14] I think it just returns false like us--checking. [22:10:17] seems like that's it. I think we need to change core to return a StatusValue. [22:10:28] Annoying deprecation period. [22:10:42] All content subclasses need to be updated :-! [22:10:46] grr [22:11:03] bool|StatusValue transitional return type? I don't like it. [22:11:35] So if you do a no-no in JSON it just says "you did it wrong" and that's it? I guess because it expects you to be using the code editor that does more specific validation for you? [22:11:58] We need a generalized class that lets you screen json content against a schema [22:13:27] Content has the methods we need, e.g. isValid [22:14:15] cos we need the same mechanism for non-wikitext, non-json content types [22:15:26] ah we do have that thing you mentioned btw [22:16:25] well it's nothing special. JudgmentValidator::validateAgainstSchema which calls justinrainbow [22:24:10] Amir1: u done for the day? I'd love a few CR comments to work with, if not [22:24:16] but it's... really late there. [22:24:30] awight: I'm doing light-weight work :P [22:24:38] haha fair enough [22:24:52] * awight imagines a weights bench treadmill desk [22:25:15] do you want me to review anything awight ? [22:25:39] only if you're still at it [22:25:51] This one is not light, but https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/461255/ [22:26:08] Most of the changes are just conforming to the new schema, so maybe not conceptually complex. [22:27:29] off-topic: I used to work out pretty much but after I got into a relationship I stopped everything, I think one day my body will come to my dream and says "We knew you didn't work out because of your health, just do some push ups once in a while to make us happy" [22:27:44] haha you got it [22:27:56] I gained 30 lbs when M had a baby [22:28:37] awight: It's rather big, is there a way to find something smaller? I will look into it tomorrow anyway [22:28:49] It's hard to understand what it changes, etc. [22:29:10] I can't rebase after the smaller ones cos they're dependent, so maybe tomorrow? [22:30:18] sure, also if you slice it to smaller ones (if possible) it would make the review super easy [22:44:28] Amir1: I can try but like I said that patch is tightly coupled because the new schema requires test fixture changes etc, it might not be possible to split. [22:45:39] if it's hard, don't worry [23:00:55] (03PS1) 10Ladsgroup: When service fails to respond, retry the job [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461815 (https://phabricator.wikimedia.org/T204753) [23:10:19] PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:34] (03PS1) 10Awight: Surface validation errors in the edit page [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461819 (https://phabricator.wikimedia.org/T197653) [23:21:06] harej: ^ :D [23:21:13] That wasn't so bad. [23:21:32] There we go [23:21:46] 10Scoring-platform-team (Current), 10JADE, 10Patch-For-Review: Surface JADE validation errors - https://phabricator.wikimedia.org/T197653 (10awight) [23:22:03] Gross, but effective [23:22:27] (03CR) 10jerkins-bot: [V: 04-1] Surface validation errors in the edit page [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461819 (https://phabricator.wikimedia.org/T197653) (owner: 10Awight) [23:23:08] (03PS3) 10Ladsgroup: Introduce ext.ores.api [extensions/ORES] - 10https://gerrit.wikimedia.org/r/459549 (https://phabricator.wikimedia.org/T201691) [23:23:10] (03CR) 10Ladsgroup: Introduce ext.ores.api (038 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/459549 (https://phabricator.wikimedia.org/T201691) (owner: 10Ladsgroup) [23:23:38] (03PS2) 10Awight: Surface validation errors in the edit page [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461819 (https://phabricator.wikimedia.org/T197653) [23:26:19] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Send metrics to grafana for non-conventional responses - https://phabricator.wikimedia.org/T205051 (10Ladsgroup) [23:39:56] (03PS6) 10Awight: Update judgment content schema [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461255 [23:39:58] (03PS3) 10Awight: Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 [23:40:00] (03PS3) 10Awight: Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 [23:40:02] (03PS3) 10Awight: Surface validation errors in the edit page [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461819 (https://phabricator.wikimedia.org/T197653) [23:40:04] (03PS1) 10Awight: Drop "append-only" support [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461824 [23:40:06] (03PS1) 10Awight: Drop page judgments for this release [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461825 (https://phabricator.wikimedia.org/T202596) [23:40:08] (03PS1) 10Awight: MCR API compatibility [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461826 [23:40:17] Amir1: After all my whinging, I found plenty to split into precursor patches... thanks for the nudge! [23:40:55] :D from 1,000 -> 400 lines [23:43:44] (03PS7) 10Awight: Update judgment content schema [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461255 [23:43:46] (03PS4) 10Awight: Use null coalescing operator [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461737 [23:43:48] (03PS4) 10Awight: Validate preferred flags [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461736 [23:43:50] (03PS4) 10Awight: Surface validation errors in the edit page [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461819 (https://phabricator.wikimedia.org/T197653) [23:43:52] (03PS1) 10Awight: Make sure we assert things in tests [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461828 [23:45:11] Nice [23:45:19] I'm about to finish my last patch for the day [23:45:29] there are several ones in review columns already [23:48:09] (03Abandoned) 10Awight: Rename namespace to NS_JUDGMENT [extensions/JADE] - 10https://gerrit.wikimedia.org/r/460615 (owner: 10Awight) [23:53:38] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, and 2 others: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10awight) Thanks for all the attention given to this, and apologies for thinking that the namespace condition would beha... [23:56:47] (03PS2) 10Awight: Include annoying copyright preamble [extensions/JADE] - 10https://gerrit.wikimedia.org/r/460611 [23:57:48] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Elisardojm) @Ladsgroup Done. [23:58:52] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Implement IP whitelisting for ORES PoolCounter - https://phabricator.wikimedia.org/T204955 (10awight)