[12:56:51] 10Scoring-platform-team, 10Wikilabels, 10Google-Code-in-2017: pytest for flask application of wikilabels - https://phabricator.wikimedia.org/T179015#3722950 (10Ladsgroup) @Florian The acceptance criteria would be to a passing ci test for at least one top level route of wikilabel based on the library I mentio... [14:13:38] o/ [14:15:24] 10Scoring-platform-team (Current), 10JADE: Post about JADE design considerations - https://phabricator.wikimedia.org/T179393#3723248 (10Halfak) [14:15:51] halfak: o/ [14:16:00] How's it going Amir1? [14:16:13] I'm a little late this morning because I needed to bring Isla to the vet for some shots. [14:16:41] Almost blew a gasket with frustration at myself because I got all of the way there and realized I'd forgotten something at home. >:( [14:16:55] 10Scoring-platform-team (Current), 10JADE, 10Documentation: Post about splitting "revision" and "edit" conceptually - https://phabricator.wikimedia.org/T179298#3723261 (10Halfak) [14:16:57] 10Scoring-platform-team (Current), 10JADE: Post about public analytics for JADE - https://phabricator.wikimedia.org/T178103#3723262 (10Halfak) [14:16:59] 10Scoring-platform-team (Current), 10JADE: Post about suppression for JADE comments - https://phabricator.wikimedia.org/T178102#3723263 (10Halfak) [14:17:01] 10Scoring-platform-team (Current), 10JADE: Post about judgments/endorsements/preference - https://phabricator.wikimedia.org/T178101#3723264 (10Halfak) [14:17:03] 10Scoring-platform-team (Current), 10JADE: Post about JADE design considerations - https://phabricator.wikimedia.org/T179393#3723260 (10Halfak) [14:17:05] 10Scoring-platform-team, 10Community-Liaisons, 10JADE: Set up working group for JADE - https://phabricator.wikimedia.org/T170954#3723265 (10Halfak) [14:17:14] 10Scoring-platform-team (Current), 10JADE, 10Epic: [Epic] Post about JADE design considerations - https://phabricator.wikimedia.org/T179393#3723248 (10Halfak) [14:17:54] 10Scoring-platform-team (Current): [Epic] Coordinate ORES UI features between Scoring and Collab teams - https://phabricator.wikimedia.org/T172743#3507900 (10Halfak) With the deployment of RC Filters for watchlists, maybe this is done? [14:18:31] :))) [14:18:39] halfak: it happens, don't sweat [14:24:10] Amir1, any thoughts on the event sourcing stuff? [14:24:14] Re. JADE [14:24:35] not yet [14:24:50] but one thing that came into my mind was using even-based architecture [14:24:59] let me find an article [14:25:35] https://en.wikipedia.org/wiki/Event-driven_architecture [14:25:42] but there are better articles [14:26:05] Right. Event sourcing ~~ Event-driven architecture. [14:28:15] Seems like a big part of the tradeoff is being forced into operating within the event-driven abstraction. [14:28:47] That means we'll likely find some pain splitting everything into events [14:30:51] This article is useful. I'd like to adopt standard terms wherever possible. [14:58:11] Amir1, ^ [14:58:39] thanks :) [15:07:56] Amir1, thoughts on this: https://www.mediawiki.org/wiki/ORES/Examples [15:08:03] Specifically the title. [15:08:05] I want to move it to a better title. [15:08:07] Like ORES/Tools [15:08:11] or ORES/Usage [15:08:19] what do you think a good name would be? [15:08:25] "examples" seems wrong to me. [15:10:12] Yeah [15:10:18] Users [15:10:32] or Clients? (Feels weird) [15:38:54] Amir1, agreed on the weird feeling. [15:39:33] Maybe ORES/Uses of [15:39:46] That's sort of a weird approach to naming. [15:39:57] hehe [15:40:07] It sounds like the beginning of a decent joke. [15:40:34] o/ awight [15:40:45] Trying to figure out what to rename https://www.mediawiki.org/wiki/ORES/Examples to [15:40:53] Because "Examples" doesn't seem right. [15:41:26] Consumers? [15:41:39] maybe too technical [16:03:53] awight : might've missed the initial discussions so asking, whats the basic motivation for using an event-driven architecture for JADE? [16:05:00] codezee: Hi! Good question, I think the primary motivation is actually to help pilot an org-wide initiative: https://wikitech.wikimedia.org/wiki/User:Ottomata/Event_Driven_Architecture [16:05:17] halfak: probably ORES/Applications for https://www.mediawiki.org/wiki/ORES/Examples [16:05:18] ? [16:05:48] But there’s also a big, practical wish: that the JADE feed is something that “third-parties” will want to consume, and they’ll be getting the same information that we get. [16:06:26] event messaging is a seemingly simple, equitable way to do that. [16:06:44] codezee: +1 “Applications” [16:08:30] codezee, I like it [16:08:47] * halfak moves [16:16:37] 10Scoring-platform-team (Current): Generate mid-level WikiProject categories - https://phabricator.wikimedia.org/T179311#3723831 (10Halfak) Looks like we don't include the top level category names yet. @Sumit said he'd like to do that in a separate PR. [16:18:08] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3723835 (10hoo) [16:23:52] RIP nose :P [16:32:32] \o/ [16:33:40] awight heh upstream just rewrote the whole layout for gerrit :). [16:33:45] and merged it [16:33:58] so i have fixed the its-phabricator plugin to use the new layout. [16:34:03] g’luck with the rebasing, eh? [16:34:08] though by its-phabricator i mean its-base [16:34:12] lololololololol [16:40:04] i suppose one downside of having the "history of states" benefit of an event driven system is that we need more space and storage right? [16:44:12] codezee: I think it’ll be linear with the number of records, so should be totally fine. [16:47:50] Heads up, I’m toggling beta ORES between revscoring 1 and 2, breakage will ensue. [16:50:11] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180#3723952 (10awight) a:05Halfak>03awight [16:52:04] codezee: I get this error in localhost too, I think you wrote the test incorrectly [16:52:04] https://travis-ci.org/wiki-ai/drafttopic [16:52:14] https://travis-ci.org/wiki-ai/drafttopic/builds/295385480 [16:52:18] Can you fix it? [16:52:25] and submit a PR? [16:52:38] halfak: Also please merge this: https://github.com/wiki-ai/drafttopic/pull/8 [16:52:45] it fixes the important part [16:56:00] Amir1, ^ failed [16:56:17] flake8 and a test failure [16:56:46] yeah but that was failing before, this shows it up [16:57:03] I don't know the codebase so I can't fix the test failure [16:57:09] but flake8 if needed [16:57:12] codezee, want to take over the PR? [16:59:31] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180#3724016 (10awight) [16:59:48] halfak: Amir1 let me check [17:00:46] Amir1: whats the command to run a pytest? [17:00:59] codezee: pip install pytest [17:01:05] and then pytest . [17:04:41] (03PS1) 10Awight: Only try pip upgrade if the wheel is present; mwtypes 0.3 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387610 [17:04:59] Amir1: halfak: ^ if you would [17:05:22] I'm so embarrassed but I can't see anything [17:05:38] haha I keep forgetting, https://gerrit.wikimedia.org/r/387610 [17:05:42] What client do you use? [17:05:49] IRCCloud [17:05:50] or did you use an IRC command to mute? [17:06:31] Amir1: Have you tried /unignore wikibugs [17:06:36] (03CR) 10Ladsgroup: Only try pip upgrade if the wheel is present; mwtypes 0.3 (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387610 (owner: 10Awight) [17:07:09] AFAIK I did [17:07:13] but did it again now [17:07:17] let's see [17:08:07] (03CR) 10Awight: Only try pip upgrade if the wheel is present; mwtypes 0.3 (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387610 (owner: 10Awight) [17:08:27] Amir1: ^ still invisible? [17:08:36] yeas :( [17:09:05] you’re spam-proof! [17:09:37] I’ll just try to remember to add you to the git reviewers, which is the better thing to do anyway. [17:10:17] https://gerrit.wikimedia.org/r/38761 [17:10:20] Woops [17:10:21] https://gerrit.wikimedia.org/r/387610 [17:10:24] Amir1, ^ [17:10:35] (03CR) 10Ladsgroup: [V: 032 C: 032] Only try pip upgrade if the wheel is present; mwtypes 0.3 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387610 (owner: 10Awight) [17:10:53] ty! [17:11:23] :) [17:11:32] I realized after the fact that this won’t come into play when rolling back, cos the scap script would be missing that line anyway. But perfectly safe. [17:18:11] Amir1: if a test makes some online request and hangs for too long, does travis wait? [17:18:35] codezee: travis has a timeout, I think it's ten minutes [17:21:50] Amir1: enchant was not found - https://travis-ci.org/wiki-ai/drafttopic/builds/295399404?utm_source=github_status&utm_medium=notification [17:22:56] codezee: Yes, you should amend your commit to my PR: https://github.com/wiki-ai/drafttopic/pull/8 [17:23:06] if you do it on top of master it fails [17:26:03] Amir1: why is flake8 complaining so much when locally it shows 0 errors? [17:27:04] it might be that you locally disabled certain errors [17:33:37] (03PS1) 10Awight: Remove the virtualenv directory each time [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387613 (https://phabricator.wikimedia.org/T179095) [17:34:26] * awight beats chest quietly. pip vs scap vs wheels will be the end of me. [17:34:29] Amir1: a blank __init__ file complains of "W391 blank line at end of file" which is absurd [17:35:52] codezee: I'd suggest to copy the file from revscoring [17:36:04] different code editors act differently [17:36:55] Amir1: Switching to a large hammer: https://gerrit.wikimedia.org/r/387613 [17:37:02] *larger [17:37:05] *hammerer [17:39:13] awight: LOL [17:39:42] Amir1: I realized you might have an ignore regex. Do you get anything listed when issuing “/ignore” without params? [17:40:24] when I create a blank __init__.py, I usually just do "touch __init__.py" [17:46:22] awight: oh I think that worked [17:46:26] let's see [17:46:37] * awight readies spam cannon [17:47:21] Amir1: the test passes locally and all files on local are on remote, what could be the possible reason for assertion failure? [17:48:42] (03PS2) 10Awight: Remove the virtualenv directory each time [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387613 (https://phabricator.wikimedia.org/T179095) [17:49:01] hargh, [17:49:05] needs a new Change-Id [17:49:06] codezee: It fails on my localhost [17:49:20] codezee: so I guess clone a clean branch and start again [17:49:32] (03PS1) 10Awight: Remove the virtualenv directory each time [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387619 (https://phabricator.wikimedia.org/T179095) [17:49:44] awight: It works [17:49:45] thanks [17:50:14] great! [17:50:41] (03PS1) 10Awight: Remove the virtualenv directory each time [services/ores/deploy] (STABLE_REVSCORING_1) - 10https://gerrit.wikimedia.org/r/387622 (https://phabricator.wikimedia.org/T179095) [17:50:51] (03Abandoned) 10Awight: Remove the virtualenv directory each time [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387613 (https://phabricator.wikimedia.org/T179095) (owner: 10Awight) [17:51:10] * awight holds severed head in hands [17:51:22] It’s these two, https://gerrit.wikimedia.org/r/#/c/387619/ https://gerrit.wikimedia.org/r/#/c/387622/ [17:51:31] I believe I did promise spam. [17:52:53] awight trick or treating are we lol :) [17:53:22] * awight splurts putrid blood at paladox [17:53:29] lol [17:54:10] :/ :/ :/ passed in a fresh copy... [17:55:37] only difference locally is python 3.6 and pytest 3.2.3 slightly above trais [17:56:24] *travis [17:57:44] is it possible to get in the failed travis workspace? [18:01:13] halfak: Amir1: hehe, I’m happy with the upgrade/downgrade at exactly the end of our window. Día de los muertos deployment it is! [18:01:26] codezee: log in? I don’t think so. [18:01:33] * halfak is in meetings. [18:01:34] WHat happened? [18:01:38] With the window? [18:01:54] halfak: precisely nothing. [18:02:04] Gotcha [18:02:18] I’m finally confident with revscoring 1<->2 toggling, though, once the last patch is mergedd. [18:02:30] So let’s plan for making tomorrow the day! [18:02:50] I’ll put it on the new cluster for fun, though. [18:03:19] codezee: I eat my words! https://docs.travis-ci.com/user/running-build-in-debug-mode/ [18:03:36] codezee: Link me the failure if you would… [18:03:39] just curious. [18:06:49] awight: i guess it passed \o/ shot an arrow in the dark but thanks anyway :) [18:07:05] (03Abandoned) 10Awight: Update to Celery 4.x [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/385102 (owner: 10Awight) [18:07:23] the issue was i had written list1 == list2 while it had to be sorted(list1) == sorted(list2) for pytest to not complain [18:07:32] though i still don't see why it passed locally.... [18:07:33] ah yeah that burns. [18:07:44] just micro-differences in python internals, I bet. [18:08:00] Amir1: the tests pass, the codecov report though complains something of no previous report found... [18:08:11] At least there was a rational answer, though. I hate suprise mystery fixes. [18:08:58] awight: the surprise is still there why it was passing locally :P maybe i had a recent python and pytest so they might've fixed this [18:09:24] eh I think it’s fair to say that [1, 2] != [2, 1] [18:09:40] but I imagine that the lists sorted the same on your local by chance [18:10:15] maybe [18:11:01] codezee: That's okay [18:11:08] I merged the first one [18:13:40] awight: something to do with mystery fixes - http://thecodinglove.com/post/166494448255/when-i-get-warnings-but-my-code-still-compiles :D [18:14:35] baahahahaha [18:14:39] ty [18:17:01] (03PS2) 10Awight: Update frozen-requirements.txt to match wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386646 [18:17:56] halfak: awight regarding yesterday's discussion on https://www.mediawiki.org/wiki/Topic:Tzvxe3sckmfoftp8 why not use the DB transaction log as a source for writing streams as aptly mentioned at - https://www.confluent.io/blog/messaging-single-source-truth/ [18:18:17] whenever a transaction fails bec of some inconsistency, the log records that [18:18:42] and rolls back [18:19:11] codezee: Yes! That’s definitely on the table, to do a “database-fronted” design with change data capture from within the db (PostgreSQL) going to our Kafka event log. [18:19:24] That has the really nice property of hiding the distributed transaction problem. [18:19:45] plus the db handles the validation part [18:20:08] My only reservation is that we can do a bit better—the event log in that case would be dictated by the database schema, and we won’t have control over it. I’d like to hand-craft the event log if possible, and make it usable and understandable to consumers. [18:20:54] codezee: I think we carried on the discussion into another meeting, sorry the only minutes were https://wikitech.wikimedia.org/wiki/User_talk:Ottomata/Event_Driven_Architecture [18:21:16] oh, i see, okay [18:21:31] But my thought there is that we can do validation with 99.something% accuracy, by using the “write-aside” state database, and keep a clean event log as the single source of truth. [18:22:20] TBH I’m not totally convinced of the event log thing, I haven’t yet understood whether schema migrations will be punishingly nasty or not. [18:22:35] awight, write that up :P [18:22:38] Mostly my own lack of experience, rather then a design thing. [18:22:51] i think for "write aside" only the issue was arising that if an event id is generated and the state is inconsistent, there's a problem, right? [18:22:53] halfak: hehe goading accepted. [18:23:40] codezee: We can get around autoincrement IDs with UUIDs, currently the concern is just a race condition where a judgment is deleted or access level changed, so an API call succeeds when it shouldn’t have. [18:24:21] The race condition might be more common than we hope, if lots of bots are acting on the realtime stream. [18:25:09] awight: so for example at 10:00 a privileged user is relegated to a non-privileged position but bec of delays the user can do some privileged operation at 10:01AM right? [18:25:19] halfak: You want to kick this one, or prefer to wait for Amir1? https://gerrit.wikimedia.org/r/#/c/387619/ [18:25:50] codezee: Exactly. They won’t actually be able to perform the action though. The weirdness is only that the API will return success. [18:25:59] ah—and we’ll have an invalid record in the event log. [18:26:57] am meetings all day :( [18:27:15] because it was write aside and once we write to the db we don't care what happened with the write? [18:27:31] halfak: no sweat! [18:28:57] codezee: Oh hey, you bring up a really important point, I think this is what halfak has been trying to tell me. If invalid events are allowed into the event log, they get propagated to all consumers. For some reason I was imagining that they would be validated during yeah db write, but in this design the db isn’t upstream any more. [18:29:01] Now I get it. [18:29:14] So we really do need to validate before letting stuff into the event log. [18:29:28] (03CR) 10Ladsgroup: [V: 032 C: 032] Remove the virtualenv directory each time [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387619 (https://phabricator.wikimedia.org/T179095) (owner: 10Awight) [18:29:34] 99.9% accuracy would be crappy. [18:29:40] Amir1: ^o/ [18:29:52] awight: yes, bec. if invalid events are in event log, db regeneration is not possible [18:30:31] It’s still possible, but consumers would need extra logic, and we might have to add “revert” events later. [18:31:23] IMO part of the requirements are that consumers should be minimalist, and shouldn’t have to e.g. know about multiple versions of the schema or other scary things. [18:31:42] awight: my doubt, does event get published before or alongside the db write operation ? [18:31:58] We’ve been debating that. [18:32:12] Ideally, there would be a distributed transaction around {db write, event log write} [18:32:28] but that’s not easy, in fact it’s almost impossible without low-level integration between the systems. [18:33:16] Even super complex retry logic in our producer wouldn’t be able to guarantee atomic both-write or both-fail, AFAIK [18:33:40] pgsql->CDC->Kafka does give us that. [18:33:54] There might be other options as well, we need to research what those might be. [18:34:24] which is why it seems sane to go through the db first then publish the event [18:34:43] One thing we can look at is whether the validity database can be smaller and trickier, even using CDC or something, but then we emit a pretty, custom-formatted event log. [18:35:01] eminently sane. Not that I should talk. [18:36:46] awight: you mentioned dependence of the event log on db schema as a drawback of Db frontend approach, can you give an example of a flexible schema scenario as advocated by the write aside design? [18:37:46] Just that, we can make the events look like anything we want in the write-aside design. They can be almost exactly the format of the API call, and very semantic. [18:38:17] In the CDC design, they’re going to be 1:1 with the database tables, AIUI. [18:46:44] there's a degree of flexibility but not too much bec. the database here is the single source of truth and hence the event logs need to represent that truth in their events too...for anyone to be able to regenerate the database at a later time [18:54:23] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3724435 (10awight) [18:57:54] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3724450 (10Ladsgroup) I have a feeling that the patch is not deployed yet. [18:59:05] and I'm still fighting with travis over wikiclass [18:59:12] and hasn't started to work on editquality [18:59:18] that will another day [19:00:09] Amir1: travis can be a bit... finicky [19:00:47] I much perfer if it was possible sadly afaik its not, to use wikimedia's ci setup instead [19:01:16] wiki-ai/wikiclass#8 (travis - 569ee45 : Amir Sarabadani): The build was fixed. https://travis-ci.org/wiki-ai/wikiclass/builds/295441272 [19:03:27] halfak: https://github.com/wiki-ai/wikiclass/pull/53 [19:03:28] o/ [19:04:29] +1'd ill wait for aaron to merge though :) [19:32:44] gotta look for internet, brb [20:52:06] (03PS1) 10Awight: Kludge a keyholder thing for deployment [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387665 [20:54:37] A few things that could use CR: https://gerrit.wikimedia.org/r/#/c/387622/ https://gerrit.wikimedia.org/r/#/c/386646/ https://gerrit.wikimedia.org/r/387665 [20:56:53] halfak: Amir1: ^ [20:59:54] (03CR) 10Ladsgroup: [V: 032 C: 032] Update frozen-requirements.txt to match wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386646 (owner: 10Awight) [21:00:32] awight: why on the old branch? We are ditching it, right? [21:00:54] That’s so that we can rollback safely—I should have explained a bit more in the comment. [21:01:14] I’ve been running through 1 -> 2 -> 1 to make sure things don’t break. They currently do break. [21:01:29] Specifically, the virtualenv dir melt down in a horrifying and unrecoverable way :D [21:01:32] *melts [21:03:42] awight: you convinced me [21:03:52] lol it was a nasty surprise. [21:03:57] (03CR) 10Ladsgroup: [V: 032 C: 032] Remove the virtualenv directory each time [services/ores/deploy] (STABLE_REVSCORING_1) - 10https://gerrit.wikimedia.org/r/387622 (https://phabricator.wikimedia.org/T179095) (owner: 10Awight) [21:04:11] That branch still has the longer timeout, I might as well clean that up... [21:04:51] awight: https://gerrit.wikimedia.org/r/#/c/387665/1 <- If Tyler gives a +1, you have my +2 :D [21:05:05] (03PS1) 10Awight: Revert "Blindly choose a timeout of 15 seconds" [services/ores/deploy] (STABLE_REVSCORING_1) - 10https://gerrit.wikimedia.org/r/387681 [21:05:45] He’s gone idle, but yes I’d like him to take a look cos of the comment character and cos it’s a weird kludge he suggested. [21:06:22] You work late, man. [21:07:04] Today is a public holiday, it's all volunteer [21:07:13] as I have a little bit of overtime too :D [21:07:15] That… that. [21:07:17] oh dear [21:07:24] (03CR) 10Ladsgroup: [V: 032 C: 032] Revert "Blindly choose a timeout of 15 seconds" [services/ores/deploy] (STABLE_REVSCORING_1) - 10https://gerrit.wikimedia.org/r/387681 (owner: 10Awight) [21:07:40] I hope you take days off from time to time. [21:07:46] (03PS2) 10Ladsgroup: Revert "Blindly choose a timeout of 15 seconds" [services/ores/deploy] (STABLE_REVSCORING_1) - 10https://gerrit.wikimedia.org/r/387681 (owner: 10Awight) [21:08:33] I will [21:32:22] (03CR) 10Thcipriani: [C: 031] Kludge a keyholder thing for deployment [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387665 (owner: 10Awight) [21:32:42] (03CR) 10Ladsgroup: [V: 032 C: 032] Kludge a keyholder thing for deployment [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387665 (owner: 10Awight) [21:33:04] Amir1: There’s something I don’t understand. Now that the extension only requests to the v3 API, why would we be getting v1 responses? Maybe these are actually malformed in another way, e.g. errors? [21:35:27] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3724798 (10awight) My guess must have been wrong. Your patches for T178962 are deployed, and we should no longer be hitting the v1 API from the extension. Now I'm... [21:35:54] OMG meetings mostly over. [21:36:02] Anything you need from me right now Amir1 and awight? [21:36:04] Gotta go do spooky things with the compact monkey. [21:36:20] Nope! Enjoy a weird evening. [21:36:31] halfak: for me just https://github.com/wiki-ai/wikiclass/pull/53 [21:36:49] I’m cutting out early, but will try to find ‘net in the morning and prep a bit more for the deployment. [21:37:20] have a good one awight :) [21:37:24] Amir1, I'm on it. [21:37:36] Maybe I’ll just set off one more deployment to get in a festive mood. [21:39:20] Amir1, got one note for you in there [22:00:03] halfak: sorry, was eating pumpkin soup [22:00:09] answered there :D