[01:59:31] ejegg: have you got a minute [02:03:38] hmm I think I know what to do actually [02:25:29] oops, sorry! [02:46:57] ejegg: no I think it's cool - I'm gonna put up a PR adding paging / more rows [02:47:31] I think the logic is OK - it kinda loads a load or rows but if the request needs more than it has it loads some more. I think there are some limitations but it's not that bad [02:58:21] cool! [02:58:52] Sorry I forgot to put my |away flag up. Sort of seamlessly transitioned from being at the computer to packing boxes [02:58:58] have a good one [03:47:02] (PS1) Eileen: Add pagination [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514205 [03:47:04] (PS1) Eileen: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) [03:48:26] :-) [03:50:16] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Add Banner Guidelines link to banner edit screen - https://phabricator.wikimedia.org/T190089... [03:50:26] Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising-Backlog, Fr-backlog-cleanup-Q3_2017-18: add tasks to #fr-backlog-cleanup-q3_2017-18 - https://phabricator.wikimedia.org/T185007 (DStrine) [03:50:34] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: URGENT: Civi Search Timing out - https://phabricator.wikimedia.org/T211062 (DStrine) [03:50:41] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Winter Wanderland, Fundraising-Backlog: Check alpha channel docs for "show stopper" issues - https://phabricator.wikimedia.org/T182032 (DStrine) [03:50:51] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog: Cancel button missing for recurring donations in Civi - https://phabricator.wikimedia.org/T202118 (DStrine) [03:51:28] Fundraising Sprint Vaporwerewolf, Fundraising Sprint Winter Wanderland, Fundraising-Backlog: Review emergency procedures for holidays - https://phabricator.wikimedia.org/T179843 (DStrine) [03:51:36] Fundraising Sprint Vaporwerewolf, Fundraising-Backlog: Support MG event in LA on the 15th - https://phabricator.wikimedia.org/T179853 (DStrine) [03:52:36] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Vaporwerewolf, Fundraising Sprint Winter Wanderland, Fundraising-Backlog, and 3 others: Amazon pay: cross-domain issue in mobile safari - https://phabricator.wikimedia.org/T180845 (DStrine) [03:52:44] Fundraising Sprint Fhabricator is spelled with an "F", Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, and 10 o... - https://phabricator.wikimedia.org/T189613 [03:52:54] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Incorrect protection reason for non-source-language version of... - https://phabricator.wikimedia.org/T213565 [03:53:03] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, and 4 others: Make protection reason localiza... - https://phabricator.wikimedia.org/T210102 [03:53:07] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Winter Wanderland, and 3 others: reassess tasks leading up ... - https://phabricator.wikimedia.org/T183290 [03:53:35] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog: Contribution Trackin... - https://phabricator.wikimedia.org/T211057 [03:53:59] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog: Delay in GC transactions making it into Civi - https://phabricator.wikimedia.org/T202079 (DStrine) [03:54:37] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and 5 othe... - https://phabricator.wikimedia.org/T186883 [03:54:54] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977 (DStrine) [03:55:00] Fundraising Sprint Vaporwerewolf, Fundraising-Backlog: Scope and create tasks for Prometheus work - https://phabricator.wikimedia.org/T179842 (DStrine) [03:55:11] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and 5 other... - https://phabricator.wikimedia.org/T171987 [03:55:29] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and 5 other... - https://phabricator.wikimedia.org/T175900 [03:55:34] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog: Spike: assess bitpay as a new processor - https://phabricator.wikimedia.org/T191220 (DStrine) [04:00:49] (PS2) Eileen: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) [05:58:34] (Abandoned) Eileen: Add pagination [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514205 (owner: Eileen) [06:57:22] (CR) Raimond Spekking: [C: +2] "false positive" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/514132 (owner: L10n-bot) [07:36:26] (PS3) Eileen: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) [07:58:30] (PS4) Eileen: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) [08:36:50] (PS5) Eileen: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) [15:20:11] mepps: thanks for the instantaneous approval on that vacation day! [15:20:30] haha no problem ejegg :) [15:58:31] (PS4) Amire80: Split apihelp messages to a separate file [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/509575 (https://phabricator.wikimedia.org/T189982) [16:16:42] (PS4) Ejegg: Switch for using contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) [16:21:18] (CR) jerkins-bot: [V: -1] Switch for using contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) (owner: Ejegg) [16:51:08] (PS1) Ejegg: Variants starting with 'upsell' mean recurrable payment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/514344 (https://phabricator.wikimedia.org/T216560) [16:51:32] Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, FR-Ingenico, Recurring-Donations, Patch-For-Review: Implement Ingenico recurring up sell - https://phabricator.wikimedia.org/T216560 (Ejegg) p:Triage→Normal a:Ejegg [17:01:46] (PS1) Ejegg: Add supportsUpsell and shouldUpsell methods [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/514351 (https://phabricator.wikimedia.org/T216560) [17:06:27] Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Epic: Render recurring upsell on result switcher - https://phabricator.wikimedia.org/T216562 (Ejegg) p:Triage→Normal a:Ejegg [17:07:34] hello fr-tech :) [17:07:40] hi jgleeson ! [17:08:14] helllllo [17:08:20] thanks for testing again the silverpop ftp access ejegg [17:08:29] at least we can get in [17:08:45] yeah, but I guess it doesn't help us much. [17:08:50] did you have to specify your IP in the silverpop console to get in? [17:08:59] So we could change the local setting to preserve more history [17:11:32] jgleeson: another thing we could check is the civi log tables [17:11:52] to see if a lot of people lost the opt-out flag between May 16th and 17th [17:12:09] or if a bunch of email addresses that were on hold changed to not-on-hold [17:12:55] if you want to go over how to check those, I'm available for a vid chat [17:13:18] or it's prob easy enough to explain in text here [17:14:39] arr, but there are a bunch of fields to check [17:14:55] log_civicrm_contact.is_opt_out [17:15:08] log_civicrm_contact.do_not_email [17:15:43] log_civicrm_value_1_communication_4.do_not_solicit [17:17:08] the join would be a bit tricky, to find rows in each of those tables that have log_action='Update' on one of the affected days, with 0 as the value for one of those fields [17:17:42] but whose previous entry for that id has a value of 1 for the field [17:17:51] so, log tables don't have a single primary key [17:18:37] their 'id' corresponds to the primary key of the corresponding data table (same name, just minus log_) [17:18:48] ah [17:18:51] but they'll have a row for each change, so the id is repeated [17:19:40] might be easier to work with temp tables than to try to find one big join to do it all [17:21:29] like, create a temp table for contact_id, update_date, new_is_opt_out, new_do_not_email, previous_change, old_is_opt_out, old_do_not_email [17:22:02] fill the first 4 columns with a select from the log table where log_date between whatever and whatever, and log_action='Update' [17:22:59] then get the previous_change with a select max(log_date) from a join between that table and the log table, where the log table's log_date is less than the update_date for the same id [17:23:33] I'm a little confused how about the import to IBM works [17:23:39] do we import from or to that system [17:23:41] and finally fill the last two columns with another join between that table and the log table, on id and previous_change=log_date [17:23:44] jgleeson both! [17:24:03] which system gets first notice of an opt-oyt [17:24:07] opt-out* [17:24:21] So, every night we send two lists that together contain every email in our system [17:24:26] jgleeson: also both :) [17:24:41] We get opt-outs from direct donor contact or from thank-you email unsubscribe links [17:25:03] and IBM's hosted opt-out page gets them from the opt-out links in CCogdill's mailings [17:25:43] our opt-out page sets the is_opt_out field. I think for some reason people using the Civi UI tend to set the do_not_solicit field. [17:26:30] but out python script (really just a wrapper around a lot of SQL commands) will take a '1' in either of those fields to mean that the donor's email address should be in the 'unsubscribe' list [17:26:38] *our python script [17:27:53] https://phabricator.wikimedia.org/diffusion/WFTO/browse/master/silverpop_export/update_table.sql$77 [17:28:04] ah right, there's also log_civicrm_email.on_hold to look at [17:28:44] anyway, that's our export to IBM [17:29:23] the import from IBM is handled by eileen's Omnimail extension to CiviCRM [17:29:41] that one calls the Silverpop APIs to get all the mailing events [17:30:04] and initially just stores them in extension-specific tables [17:30:30] but then has an async job to go through those tables and translate the events into Civi-native records [17:30:38] so you can see them via the CiviMail UI [17:31:04] I believe there's one job that transcribes mail send events [17:31:16] and another job that transcribes opt-out clicks [17:32:25] and yet another (the newest) that imports completely new contacts from IBM - these are the people that signed up through the 'remind me later' form [17:36:49] jgleeson: anyway, I'd be pretty surprised if any of the jobs that import from IBM were the culprit [17:37:05] the only one that touches one of the unsubscribe fields always turns it on, not off [17:37:55] do the log tables sound like a good next step to you? [17:39:48] just rereading, I got distracted [17:43:08] so would the hope be that the tmp table entries would confirm the decline? [17:43:49] jgleeson: right, and also tell us which field saw all those changes, and when [17:44:17] from those data points, we can maybe figure out if it was a job that flipped them, an import, or what [17:49:01] those log_ tables also have a log_conn_id column [17:49:20] which should be the same for all the changes made in a single request, or a single job process [17:49:44] and that can help track down what the root cause was [17:49:54] ok sounds like a plan, I'll give it a go [17:50:02] rockin! [17:50:21] thanks for the detailed overview! [17:52:03] any time :) [17:56:02] (PS2) Ejegg: Add supportsRecurringUpsell and showRecurringUpsell [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/514351 (https://phabricator.wikimedia.org/T216560) [18:01:46] ejegg what was the ingenico console I didn't have access too, the payments console? and what is the other one called [18:02:39] cstone yeah, I think the old one is the payments console [18:02:47] fundraising-tech-ops, DC-Ops, Operations, decommission, ops-eqiad: decommission frav1001.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T222109 (RobH) [18:02:59] and the newer one (where you do have an account) is the configuration console [18:03:13] ok cool thanks! [18:03:22] MBeat: can you help get cstone access to the wpc payments console at ingenico / globalcollect? [18:03:36] sure thing [18:03:42] cstone I'm not sure what wpc stands for, it's just in the URL [18:03:48] hah okay [18:03:49] thanks! [18:03:58] thanks MBeat ! [18:04:11] gonna go make some lunch, back in a bit [18:36:29] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (AndyRussG) >>! In T224733#5230636, @fdans wrote: > @Jgreen can you provide a little more context if you think we could be of help? Hi @fdans! These... [18:37:11] hey XenoRyet Ejegg let me know if it would be helpful for me to show you how to access the FTP files in the IBM UI (re https://phabricator.wikimedia.org/T223935#5226483) [19:15:11] ccogdill: oh, cool! I think I might have stumbled across that, but since I didn't see any files there I kept trying via the command line. [19:15:30] Then when I finally got in that way, there were no files in upload after all [19:15:41] the foldering system is super confusing but I THINK they are there [19:17:18] ah crap, looks like we only have the files going back to 5/21 :( [19:17:36] so we retain for 2 weeks, it seems [19:18:10] ccogdill oh really? Shoot, I guess I had the wrong place in the UI. In that case, I'd love to have a tour when you're free! [19:18:54] fr-tech has our standup in 10 min - if you want to show the whole team I'll send you the link [19:19:28] hmm, on an unrelated not it looks like we just had some db hangs in civi [19:19:32] *note [19:20:16] I have a meeting in 10 [19:20:19] maybe next week? [19:21:23] ok, cool [19:23:09] well, no big processes thumping the db just now [19:24:18] ejegg, I was just scratching my head why my tmp queries were hanging [19:24:40] on frdev1001? [19:24:42] jgleeson_: you're doing those on frdev though, right? [19:24:46] yeah [19:24:49] That shouldn't affect the main db [19:25:08] ah cool [19:25:12] I just checked using the drush trick to become the civi sql user [19:25:18] then 'show full processlist' [19:25:26] and nothing's hanging up the primary server [19:25:27] yup I'm doin that in my other tmux window [19:25:30] currently [19:25:58] wow [19:26:01] this query [20:08:40] fundraising-tech-ops: Backup strategy for Grafana - https://phabricator.wikimedia.org/T217353 (cwdent) a968e43d enable grafana backups b4ecac3f pull grafana archives to backup server [20:13:09] fundraising-tech-ops: Backup strategy for Grafana - https://phabricator.wikimedia.org/T217353 (cwdent) ef920db allow loggers to ssh to monitoring servers for archive collection [20:13:40] fundraising-tech-ops: Backup strategy for Grafana - https://phabricator.wikimedia.org/T217353 (cwdent) Pretty sure everything is done here, I will close once the archive sync runs successfully. [20:14:42] Fundraising-Backlog, fundraising-tech-ops: investigate which civi job is creating/deleting tables right during the nightly database backups - https://phabricator.wikimedia.org/T225022 (Jgreen) [20:15:52] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (Jgreen) >>! In T224733#5234505, @AndyRussG wrote: >>>! In T224733#5230636, @fdans wrote: >> @Jgreen can you provide a little more context if you thin... [20:17:19] Jeff_Green: https://phabricator.wikimedia.org/T224733 is about the new pipeline? [20:17:32] Which log files are they? So maybe we could compare with log files for the old pipeline? [20:17:39] oh good point [20:17:50] The new pipeline is not currently getting anything into the db [20:18:19] this is the landingpage-impressions.*.json.log files [20:18:28] hmmm [20:18:35] and which is the old pipeline files? [20:18:47] maybe we could mention the filename there on the task? [20:18:50] those are on americium [20:19:24] the old pipeline is: [20:19:31] landingpages.tsv.*.log [20:19:42] those appear to be flowing [20:20:22] centralnotice impressions (also on alnitak, new pipeline) may be impacted too now [20:20:55] Jeff_Green: ok hmmm [20:21:33] thanks much, yeah let's put all of the above on the task? [20:22:02] ok [20:22:41] Jeff_Green: thx!!!!!!!! [20:23:16] thinking [20:31:53] ejegg: is it worth pushing through the changes to the Super-duper that are on staging before show & tell in 30 mins [20:32:54] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (Jgreen) More context/clarification, we currently have two kafkatee pipelines which I'll refer to as "old" vs "new". The old pipeline is running on a... [20:37:01] eileen: I'll take a quick look at the code. I think I have an idea of what you were hoping to improve [21:01:35] (PS3) Ejegg: Add supportsRecurringUpsell and showRecurringUpsell [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/514351 (https://phabricator.wikimedia.org/T216560) [21:04:10] (CR) Ejegg: [C: -2] "Thanks for calling our attention to this bit of cruft! It turns out we weren't using that function anyway, so I've removed it completely i" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/512458 (owner: Jforrester) [21:08:30] (CR) Jforrester: [C: -1] "Even better. :-)" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/512458 (owner: Jforrester) [21:08:32] (Abandoned) Jforrester: Replace references to deprecated Squid config with modern CDN ones [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/512458 (owner: Jforrester) [21:12:48] (PS1) Jforrester: Replace references to deprecated Squid config with modern CDN ones [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/514396 [21:23:29] (CR) jerkins-bot: [V: -1] Replace references to deprecated Squid config with modern CDN ones [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/514396 (owner: Jforrester) [21:33:25] (CR) Ejegg: [C: +2] "Looks good! Maybe rename removedMergedMatch to just removePair since it's got a new use?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) (owner: Eileen) [21:37:11] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (Nuria) @Jgreen I think you might want to get in touch with @AndyRussG t Both the kafka topics you are filtering are being persisted to hive in tabl... [21:37:58] (Merged) jenkins-bot: Add pagination and 'later' button to dedupe screen [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/514206 (https://phabricator.wikimedia.org/T217903) (owner: Eileen) [21:43:36] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add a tool for identifying privacy donor template emails - https://phabricator.wikimedia.org/T225032 (DStrine) [21:49:20] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: user testing notes for the new deduper feature - https://phabricator.wikimedia.org/T225033 (DStrine) [21:52:22] ugh, more failmail [21:53:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: script for name equivalents - https://phabricator.wikimedia.org/T225034 (DStrine) [21:53:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: civi dedupe script for name equivalents - https://phabricator.wikimedia.org/T225034 (DStrine) [21:53:49] hmm, looks like an unlimited dedupe query again [21:53:54] joining email to email [21:54:18] MBeat: I just killed a dedupe query of yours [21:54:41] woops, ty ejegg [21:56:00] was testing some things w/ Eileen, was not a typical query we’d run so hopefully it does not recur [21:57:18] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/514403 [21:57:37] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/514403 (owner: Eileen) [21:58:21] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/514403 (owner: Eileen) [22:04:14] ejegg: ug [22:04:39] fr-tech is there tech-talk today? [22:04:46] oops, be there in a sec! [22:04:52] whoops same [22:06:48] !log civicrm revision changed from 506ebe2f2a to 5c02e62d6e, config revision is 63438eea43 [22:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:58] MBeat: I just deployed that update [22:21:55] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (Ottomata) The events are not in Hive: ` select year, month, day, count(*) from LandingPageImpression where year=2019 and month=5 and day in (27,28,29... [22:39:07] ty eileen [23:08:36] Fundraising-Backlog, fundraising-tech-ops, Analytics: investigate recent divot in landingpage log activity - https://phabricator.wikimedia.org/T224733 (Ejegg) Possibly-related commit in the FundraiserLandingPage extension: Remove temporary EventLogging code-duplication rEFUL9f0535ff0750ee605eede776fa... [23:10:31] AndyRussG: Have you looked at the recent EventLogging changes? Looks like the schema version is now specified in an 'attributes' bit of extension.json [23:11:20] ejegg: I haven't... However we don't use standard EL client-side code [23:11:29] We should though, since that has been improved recently too [23:11:33] we do after https://phabricator.wikimedia.org/rEFUL9f0535ff0750ee605eede776facc83cf0f32bd44 [23:11:52] which was merged May 22 [23:12:01] pretty contemporaneous with that divot [23:12:18] (or at least the major one - wouldn't explain the shorter outages earlier) [23:13:01] I didn't realize till today that the outage was in the new stream, rather than the old one [23:13:14] I guess that explains why fr-not-tech hasn't been concerned [23:13:17] ejegg: ohhh interesting! Sorry I hadn't followed that change in LandingPage ext [23:13:31] Yeah I thought it was the old pipeline too [23:13:43] I'm just trying to understand the implications [23:13:51] so far have only found a loose correlation [23:14:33] Hmmm interesting. Previously you did set a schema version number in extension.json but that was only if you used the standard client-side validation [23:14:56] Currently also no validation client-side though... Also, that may have changed [23:15:32] ooh, could we be running afoul of validation in a way that the server side was letting through? [23:17:57] gotta wash some dishes, might pick up this thread later [23:56:46] ejegg it should have been passing validation previously. We sent the schema version number in the hacky-code-generated event that we were sending [23:57:02] If it didn't pass validation, it ended up in a different Kafka stream