[14:09:53] o/ [14:38:26] halfak: o/ [14:38:34] I started to work on eswikibooks damaging [14:38:44] Amir1, great! [14:38:55] Do you know the status of extension stuff in beta? [14:41:26] awight, ^ [14:41:46] I was hoping we would be talking about deploying in prod right now. [14:42:45] o/ [14:42:49] Not awesome [14:42:54] I left a comment in the task [14:43:06] It’s still working, but I can’t explain why [14:43:11] & seems to be uncached [14:43:25] halfak: ^ [14:44:28] awight, I'm confused. Are we blocked on someone else? or is there some insurmountable problem? [14:44:44] Also, I haven’t tested against the new-thresholds endpoint yet. [14:44:47] Blocked? [14:44:51] I ran out of time in the day is all [14:45:27] Gotcha. [14:45:55] Unfortunately, I couldn’t find any of the extension log messages in logstash-beta [14:46:01] that… didn’t help the debugging [14:46:06] * halfak continues nervousness about the delay [14:46:16] Also couldn’t manage to guess the memcache key to debug the cache. [14:46:34] yeah it makes me nervous too, maybe we should revert on master. [14:47:52] "revert on master"? [14:48:26] I’m not up to speed on our deployment fu yet, but that’s usually the protocol, if there’s a danger of the train rolling out broken code to production. [14:52:55] sheesh, I should have checked that schedule. 1.31.0-wmf3 might have caught my extension changes, checking now. [14:55:17] I only see a 1.31.0-wmf2 branch, and it was cut before my stuff was merged. But I wish I knew where the new branch was. [14:56:05] Still not seeing it in git, but: https://www.mediawiki.org/wiki/MediaWiki_1.31/wmf.3#ORES [14:56:07] looks safe. [14:56:58] awight: the branch will go to all wikis expect wikipedia (expect cawiki and hewiki) tonight [14:57:21] Amir1: I think it’s okay though, since my sketchy changes aren’t included. [14:57:43] no, both are included [14:58:08] * awight still downloading the mw-core changes to check that [15:00:15] weird, test.wp is still on wmf2 althought the roadmap says it should be on wmf3. [15:00:31] & I don’t see a wmf3 branch in core at all. [15:00:51] Asking in -releng... [15:02:27] Amir1: Where are you seeing that the patches are included? [15:02:55] awight: it's probably because the train stuck [15:03:14] https://phabricator.wikimedia.org/T174359 [15:04:43] K so we got “lucky” and need to revert ASAP is what I’m thinking. Amir1? [15:05:20] I'm so confused. Is some of our code going to production before it is merged in gerrit? [15:05:28] And deployed in beta [15:05:40] halfak: it's already in beta [15:05:49] halfak: we merged my two patches to the extension yesterday [15:05:55] unless beta is broken but that's hardly happens [15:05:59] *that [15:06:07] beta does have the new changes. [15:06:12] Oh! So I can deploy to beta! [15:06:23] awight: why we need to revert the patch out of wmf.3? [15:06:30] halfak: It’s already deployed. [15:06:38] ORES isn't [15:06:41] or is it? [15:06:49] the extension or the service? [15:06:50] Amir1: https://phabricator.wikimedia.org/T175053#3673985 [15:07:12] halfak: Extension:ORES is already deployed to beta, with my compat changes. [15:07:23] But is the ORES server [15:07:24] ? [15:07:28] no [15:07:35] * halfak starts that right now [15:07:38] * halfak ignores his meeting [15:07:44] Let's see if it blows up! [15:07:56] wait wait [15:07:59] pls don't [15:08:05] it seems it's already have problem [15:08:11] It needs to run correctly in this configuration before anything else. [15:08:21] Oh. OK [15:08:44] that's why Adam suggests we need to cherry-pick it out of the current deploying branch [15:08:51] … where the wiki has new extension code that includes forwards-compat, but hits an old thresholds API. [15:11:06] (03PS1) 10Awight: Revert "Fallback to old thresholds API as necessary" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383593 (https://phabricator.wikimedia.org/T175053) [15:11:08] (03PS1) 10Awight: Revert "Support new thresholds API" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383594 (https://phabricator.wikimedia.org/T175053) [15:11:28] Apologies for not thinking of this last night, when I realized it wasn’t safe to deploy. [15:11:49] The big problem is that I can’t prove to myself that it’s caching the thresholds. [15:12:18] In fact, it seems to not be attempting to use the new thresholds API syntax at all, which is an utter mystery to me. [15:12:33] I don’t see any way that would happen, looking at the code. [15:13:05] but since I was unable to find the logs, I’m just flying into a snowbank on instruments. [15:13:21] hmm, or lack of instruments [15:14:44] (03CR) 10jerkins-bot: [V: 04-1] Revert "Fallback to old thresholds API as necessary" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383593 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [15:14:49] O_O [15:14:53] oh boy [15:15:42] (03CR) 10Awight: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383593 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [15:18:05] lol I’m not trying to rub salt in my own wounds, but should mention that when I say “ran out of hours in the day”, I actually had to miss 2 hours yesterday, which will probably turn into time off request. [15:20:14] halfak: Amir1: The reverts are in CI, please merge once they have V+2, if you agree with my assessment…. https://gerrit.wikimedia.org/r/383593 https://gerrit.wikimedia.org/r/383594 [15:21:24] (03CR) 10Ladsgroup: [C: 032] Revert "Fallback to old thresholds API as necessary" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383593 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [15:21:33] (03CR) 10Ladsgroup: [C: 032] Revert "Support new thresholds API" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383594 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [15:21:49] awight: please cherry-pick these reverts into wmf.3 [15:22:56] I don’t think there’s a branch yet [15:23:39] Amir1: ty for the merges! [15:24:25] yeah, it seems so [15:29:49] (03Merged) 10jenkins-bot: Revert "Fallback to old thresholds API as necessary" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383593 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [15:29:52] (03Merged) 10jenkins-bot: Revert "Support new thresholds API" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383594 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [16:03:32] ping Amir1 [16:03:36] n/m [16:03:47] :D [16:28:14] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3676210 (10Halfak) Maybe rerun test with `sudo lsof | wc -l`. There's an argument to limit to process ID. [16:56:05] 10Scoring-platform-team (Current), 10ORES, 10Wikilabels, 10revscoring, and 2 others: [Discuss] Moving from nosetests to pytest - https://phabricator.wikimedia.org/T174660#3676303 (10Halfak) Talked to @Ladsgroup. He's going to present this at an upcoming Sync meeting so we can discuss the switch. [17:00:09] Amir1: Thanks for the pointer, I’m on deployment-fluorine02.eqiad.wmflabs:/srv/mw-log now. [17:00:48] Woot! [17:01:06] And there’s the explanation for the failure, just about to scroll off the top of exception.log [17:01:39] grr. different failure I didn’t know about yet. [17:03:08] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10ORES, 10MW-1.31-release-notes (WMF-deploy-2017-10-10 (1.31.0-wmf.3)), 10Patch-For-Review: Make RCFilters compatible with both the old and new thresholds APIs - https://phabricator.wikimedia.org/T175053#3676355 (10awight) JobRunner exceptio... [17:11:41] 10Scoring-platform-team: FetchScoreJob is trying to update scores for nonexistent models - https://phabricator.wikimedia.org/T177967#3676372 (10awight) [17:13:55] 10Scoring-platform-team: FetchScoreJob is trying to update scores for nonexistent models - https://phabricator.wikimedia.org/T177967#3676384 (10awight) Probably unrelated, Beta Redis needs some more configuration? ``` 2017-10-11 06:52:30 [fdf9ae2aefff4a3a78b47ac5] deployment-videoscaler01 wikidatawiki 1.31.0-al... [17:19:22] YES. Amir1 thanks again, I found our loglines. [17:19:35] yw [17:19:44] and: bahahaha looks like it was working exactly as hoped. [17:19:45] <3 [17:34:18] Amir1: This is embarrassing, but I think we should redeploy. The logs are perfect. [17:34:53] awight: do you know it's hitting the server too many times? [17:35:31] Amir1: The main code is fine, I can see all the outbound calls and it’s just 1x minute. [17:35:39] Good question though, *why* [17:35:48] Lemme check the IPs [18:03:07] Logstash is amazing. But I hate it. [18:03:18] Cannot manage to search for anything at all that relates to what I’m looking for. [18:03:42] for instance, “model_info” seems to get split on the underscore. 1.31.0-alpha (UA) doesn’t seem to do anything. [18:03:45] lovely [18:09:41] https://github.com/wiki-ai/wikilabels/pull/200 [18:09:56] awight: halfak ^ [18:10:12] nice, reading now. [18:13:31] (03PS1) 10Awight: Support new thresholds API (take 2) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383625 (https://phabricator.wikimedia.org/T175053) [18:13:32] Let’s just… [18:13:33] (03PS1) 10Awight: Fallback to old thresholds API as necessary (take 2) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383626 (https://phabricator.wikimedia.org/T175053) [18:13:35] ^ [18:14:35] 10Scoring-platform-team, 10Wikilabels: Oauth login does not return the user to the previous page - https://phabricator.wikimedia.org/T175724#3601518 (10Ladsgroup) https://github.com/wiki-ai/wikilabels/pull/201 [18:14:49] 10Scoring-platform-team (Current), 10Wikilabels, 10User-Ladsgroup: Oauth login does not return the user to the previous page - https://phabricator.wikimedia.org/T175724#3676606 (10Ladsgroup) a:03Ladsgroup [18:21:02] (03CR) 10jerkins-bot: [V: 04-1] Support new thresholds API (take 2) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383625 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [18:26:49] (03CR) 10Awight: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383625 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [18:30:16] out for an hour... [19:19:16] halfak: it might be that you haven't cleared the cache (ctrl+shift+r), did you? [19:19:24] Yeah I did. [19:21:17] so weird. I'm going to check on some more things. [19:21:22] that's weird [19:22:55] I even have the problem in an incognito window [19:24:16] I confirmed that my local session seems to be working [19:28:31] I've confirmed that I am running new code. [19:29:49] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Wikidata: Support ORES for propagated Wikidata edits - https://phabricator.wikimedia.org/T158025#3677119 (10jmatazzoni) [19:29:58] I've confirmed that /auth/initiate is called without argument [19:31:27] OHHHH! I'm not getting a window opened. It's all running in the current tab. [19:35:58] It looks like it is supposed to open a new window [19:36:06] Amir1, are you getting a little popup for oauth? [19:36:10] nope [19:36:34] that was weird for me too [19:37:10] hmm... I have to put this down to get back to work on file handles. Maybe awight can try. [19:37:32] FWIW, I can figure out why it isn't working. I can clearly see the right code being loaded by my browser. [19:37:33] okay, let me see why it happens and I will let yo uknow [19:37:34] :/ [19:38:17] halfak: OMFG, I found out what's wrong [19:38:22] I will fix it right now [19:38:44] \o/ [19:48:47] o/ Anyone feel like re-merging my extension patches? [19:48:59] Amir1: ^ If you don’t mind [19:49:16] awight: sure :) [19:49:16] https://gerrit.wikimedia.org/r/#/c/383625 [19:49:23] https://gerrit.wikimedia.org/r/#/c/383626 [19:49:48] thanks! [19:50:57] (03CR) 10Ladsgroup: [C: 032] "Re-reverting" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383626 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [19:51:05] (03CR) 10Ladsgroup: [C: 032] "Re-reverting" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383625 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [19:51:18] awight: logstash works like google (used to), if you want verbatim search, you should put things in quotes [19:51:58] tgr: That didn’t seem to work either… [19:52:19] e.g. “Falling back to old threshold stats” [19:52:31] pretty sure it does [19:52:46] It’s logged at INFO level, shows up all over beta wfDebug.log [19:52:50] unless it's prcessed somehow before being sent to logstash [19:53:15] (03Merged) 10jenkins-bot: Support new thresholds API (take 2) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383625 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [19:53:19] or not sent to logstash at all; we have an open ticket about somewhere [19:53:24] (03Merged) 10jenkins-bot: Fallback to old thresholds API as necessary (take 2) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383626 (https://phabricator.wikimedia.org/T175053) (owner: 10Awight) [19:53:29] although I think it only happens to fatals [19:53:58] hmm there *is* an exception involved, but it’s caught and logged normally AFAICT [19:56:57] halfak: I fixed it [19:59:21] let me do it in a better way [20:00:05] OAuth is fun’t [20:05:48] yeah it's not, ://// [20:05:57] halfak: made another change, can you try again now [20:05:59] please :D [20:06:41] Amir1, will do shortly. [20:06:49] Looks like I've narrowed down our filehandle issue. [20:07:07] halfak: pray tell [20:07:08] Looks like we're not doing anything crazy, but celery opens a ton of FIFO pipes [20:07:09] I need to leave, it's 10 PM already I'm in the office and Berlin is not super safe (not to mention I'm starving) [20:07:25] And it grows superlinearly with the # of workers. [20:07:38] Amir1: good luck! [20:07:39] Amir1, no worries. I'll push a change if I need one. [20:07:41] o/ [20:07:45] Have a good night! [20:07:45] I will catch up from home [20:07:50] see you tomorrow [20:08:04] halfak: I need to contact you about my work in telegram [20:08:13] o/ [20:09:12] kk [20:10:50] awight, generally, I think we can just bump the limit and not worry about it. But I want to get a good estimate of how many FIFO pipes we can expect. [20:11:11] awight, how many workers should we have per node? [20:11:20] In the ORES stress test -- currently. [20:17:09] Looks like 480!!! [20:17:12] Woah [20:17:13] OK [20:18:28] * halfak is going to crash his computer. BRB [20:18:28] ;) [20:20:17] halfak: The thing is, something slowly creeps up and eventually hits the limit. If that’s the case, then no limit will be high enough. [20:20:45] One thing we should do is run a long stress test and plot the number of file handles at regular intervals. [20:20:56] awight, not sure about the creeping but I do know that it takes a little while to get all of the workers started. [20:21:04] awight, +1 [20:22:53] * halfak forkbombs himself [20:23:51] OSError: [Errno 24] Too many open files [20:23:53] aha! [20:29:21] “woohoo!" [20:30:40] I don’t get why the file descriptors would increase at all. The workers have already been started, and their setup is done before forking. [20:33:33] o/ [20:33:50] Hey Zppix [20:44:09] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): Very long search times on RC Page for "Very likely good faith" + "Likely have problems" (on en.wiki only?) - https://phabricator.wikimedia.org/T164796#3245687 (... [20:46:59] (03PS1) 10Awight: [DNM] Revert "Fallback to old thresholds API as necessary (take 2)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383667 [20:49:49] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3677535 (10Halfak) ``` $ lsof -p 19546 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python3 19546 halfak cwd DIR 2... [20:50:24] halfak: I’m going to update ores-beta.wmflabs.org to revscoring 2.0 [20:50:52] That lets us test the other half of the thresholds API fallback conditional, using new syntax. [20:51:27] Note that the config doesn’t need to change, we transparently munge old syntax formulas. [20:56:27] 10Scoring-platform-team, 10Wikilabels, 10User-Ladsgroup: Add list of labelers to campaign stats (sort by labels submitted) - https://phabricator.wikimedia.org/T178004#3677563 (10Halfak) [21:13:05] halfak: Here’s a thought: we launch the ores cluster with a known deficient number of workers, as long at it’s enough capacity to serve all production requests + a bit. [21:14:41] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3677643 (10awight) Looks this is a known thing that the number of connections between each process, c.f. the "handshake problem", if each process... [21:18:15] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): Very long search times on RC Page for "Very likely good faith" + "Likely have problems" (on en.wiki only?) - https://phabricator.wikimedia.org/T164796#3677651 (... [21:25:47] halfak: According to the theory that I think we’re validating in T177036, if we fix Celery code to immediately close sibling pipes in each child, then we won’t be grazing the fileno limits. [21:25:48] T177036: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036 [21:26:31] awight, not sure that's better than just upping the limit. [21:27:15] oh right, that’s better if we can afford it. [21:27:54] On the task I guesstimated that the total pipe count will grow to N^2 until we fix the leak. [21:28:30] That accounts for why a warm cluster at 1/2 the fileno limit crashed a few minutes into heavy stress testing. [21:28:53] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3677669 (10Halfak) I just bumped my ulimit to be able to start up a dev instance with 480 workers. I got ``` $ lsof -p 4516 | grep FIFO | wc -l... [21:29:09] hmm, actually I can’t explain why the workers crash in the first place, this only explains the overall FH exhaustion. [21:30:27] workers crash when FH is exhausted [21:30:53] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3677676 (10awight) By the N^2 guess, we would need 480 x 480 = 230,400 file handles. Cool, so let's take the max of our guesses and try a limit... [21:31:37] halfak: but the FHs shouldn’t be exhausted by workers doing regular work, only by them crashing and being restarted with a higher number of ghost children. [21:32:06] awight, we don't yet have evidence of a leak [21:32:13] It takes a long time to start up 480 workers. [21:32:30] OK lemme do a stress test where I measure all the things we care about. [21:32:36] +1 [21:32:54] I suggest picking on a worker process and tracking it's FIFO handles. [21:32:56] https://etherpad.wikimedia.org/p/ORES_stress_test_variables_to_monitor [21:37:03] 10Scoring-platform-team, 10ORES: Clean up file handle and Redis connection management in ORES worker and celery processes - https://phabricator.wikimedia.org/T177036#3677696 (10Halfak) Based on previous estimates, it looks like we open about 173 file handles for each worker -- mostly for *.so files. So I'd... [21:37:33] Just bumped my estimate to 1,237,132 to include the non-FIFO file handles. [21:37:51] Nice work eliminating the .so's! [21:39:45] I'm currently using 57.5 GB of the 16GB of memory on my laptop [21:39:46] :D [21:39:55] Damn RES estimates. why are you so wrong? [21:40:02] hehe could be swapping though [21:40:09] Not using swap at all [21:40:17] it is a shocking thing to learn that RES is not a thing. [21:40:23] only have 8Gb of it anyway [21:40:30] I totally agree. [21:41:06] When we finish the FIFO nonsense, we should try that /proc tool to measure actual memory sharing. [21:44:31] Right. 'cause right now, we're winging it. [21:45:42] halfak: Are you making that task to ask akosiaris to increase fileno limits again, or should I? [21:45:54] awight, please do [21:45:59] cool. [21:46:06] I'm rushing to finish brazil visa stuff now. [21:46:16] Sounds important [21:46:39] I guess we can’t do the next stress test until the limit is raised. [21:46:39] They want a damn notarized bank statement -- scanned and emailed. WHAT WOULD THE MEAN!? [21:46:47] *that even mean [21:46:59] lol that you are good at Adobe [21:47:12] 1. How is the notary going to verify my bank statement [21:47:17] An impossible document that you *shouldn’t* be able to produce. [21:47:37] 2. How is it that I'm going to scan a notary stamp and they are going to be sure I didn't fabricate anything? [21:55:11] halfak: hey, other thing: [21:55:24] Mind if I deploy revscoring 2.0 to ores-beta.wmflabs? [21:55:39] * awight checks backscroll [21:57:38] awight, no problem. We might want to deploy and undeploy it a few times to make sure the transition happens as expected. [21:57:47] No worries by me though. Deploy there when ready. [21:58:00] Oh one sec. We need to update that repo. [21:58:03] I'll do that quick. [21:59:42] halfak: excellent idea to test the rollback. [22:00:17] oh, ty! Feel free to leave that for me. [22:02:13] No worries. Should be quick. Just some updates. [22:04:34] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, 10User-Ladsgroup: Review and fix file handle management in worker and celery processes - https://phabricator.wikimedia.org/T174402#3677795 (10awight) @akosiaris We're now estimating that 480 workers will eventually use 220k–1.2M... [22:07:34] (03PS1) 10Halfak: Bumps submodules for models based on revscoring 2.0.8 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/383716 [22:07:39] Amir1, https://gerrit.wikimedia.org/r/383716 [22:07:41] Woops. [22:07:43] awight, https://gerrit.wikimedia.org/r/383716 [22:07:44] hgehe [22:08:13] * halfak runs away! [22:08:19] Have a good one! [22:08:26] halfak: erm one thing— [22:08:37] is that server deployed like other beta things, as soon as master merges? [22:08:55] * awight reviews https://wikitech.wikimedia.org/wiki/ORES/Deployment [22:09:16] halfak: nvm, the docs are good. [22:09:32] (03CR) 10Awight: [C: 032] Bumps submodules for models based on revscoring 2.0.8 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/383716 (owner: 10Halfak) [22:11:13] (03CR) 10Awight: [V: 032 C: 032] Bumps submodules for models based on revscoring 2.0.8 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/383716 (owner: 10Halfak) [22:14:14] \o/ for docs [22:50:16] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): Very long search times on RC Page for "Very likely good faith" + "Likely have problems" (on en.wiki only?) - https://phabricator.wikimedia.org/T164796#3677980 (... [22:58:13] RoanKattouw: Since you were asking, I’m currently trying to deploy revscoring 2.0 to the beta ORES service. [23:25:15] awight: OK cool [23:39:38] RoanKattouw: Might be blocked by an inexplicable git-ssh.wmo timeout, I’ll keep you posted at the end of day.