[00:02:16] AndyRussG: oh hey, it's Apache Traffic Server now I think [00:04:08] ejegg: no they reverted to Varnish [00:04:25] ATS is only on a slightly more inner layer now [00:04:30] oh wow, that was recent, i guess. [00:04:52] i'm reading the nov 25 2020 article about them switching to ATS [00:05:46] ah, i see, 'replacement of Varnish with Apache Traffic Server (ATS) as the on-disk HTTP cache component of the CDN. [00:05:49] ' [00:06:43] ejegg: if u like in a bit I can send here the to the log from my discussion MOnday with bblack on -traffic [00:10:02] ah thanks, just found the link to that channel's logs [00:17:18] :) [00:19:26] hmm, sounds like in the easiest case it's still a performance hit to need to do ESI combination of 2 blobs on every load where they're now just delivering a wholly rendered single blob [00:19:30] replacement of Varnish with Apache Traffic Server (ATS) as the on-disk HTTP cache component of the CDN. [00:19:43] grr, that's not what I wanted to paste [00:19:51] [22:54:59] but the issue we now face is that our edge performance doesn't scale well, because basically-all pageviews are now ESI-rendered from two cache objects, and have to be combined on the fly as we output them. [00:20:17] ejegg: yes.... but it's still the simplest option [00:20:35] and there are other uses cases for ESI combining blobs, and this does go in the direction they've wanted to go in [00:21:06] so basically for Traffic it'd be only that bit, and no special ESI code, just fetch a cached blob and inject it [00:21:25] and for CN only minimal code changes [00:23:23] right, like an update to Special:BannerLoader to consider a special cookie [00:23:55] and then the traffic team treats Speical:BannerLoader as explodable based on that cookie value [00:24:17] yup yip [00:24:47] so banner chooser logic then sets a cookie instead of making the request [00:25:10] hmm, any post-banner js logic gets interesting [00:25:24] just the same as now [00:25:24] I guess all of that can run on the next pageload? [00:25:39] yep! [00:26:06] ok, so it basically reads the cookie that has just been set, to know what banner has been chosen [00:26:14] then I guess unsets the cookie [00:26:24] and the next fresh pageload JS runs [00:26:31] to decide the next banner [00:27:58] err rather, after the post-banner JS for banner 1 which is currently showing in the page, we then re-run the chooser logic based on the new localStorage state to decide what banner to show on the next pageload and set that cookie [00:29:04] we wouldn't want to deploy this while still relying on the BannerLoader web logs to get an impression count, I guess [00:29:24] so getting EventLogging-based impression counts working is a prerequisit [00:29:27] e [00:30:02] ejegg: we don't rely on banner loader weblogs [00:30:13] we have our own beacon/impression thing still going [00:30:28] and that could still run exactly as is, no changes, or very few, I think! [00:30:40] oh, doesn't kafka basically feed us those web logs? [00:30:53] but not bannerloader [00:32:52] ah right, beacon [00:33:03] ok, ok [00:34:25] so how does this affect our ability to take campaigns up and down? [00:35:13] If Special:BannerLoader isn't considering campaign name, it doesn't know to blank the content out when the campaign is disabled [00:35:57] one hour tests get pretty smeary too [00:37:19] Not trying to sink the idea by any means [00:37:38] it's definitely the most workable i've heard so far! [00:37:50] just trying to anticipate all the effects [00:39:01] ok, so maybe Special:BannerLoader can consider campaign name. it's not really a #campaigns * #banners explosion because each banner is probably only in a couple different campaigns [00:51:35] ok, gonna reply to that mini-thread you just started AndyRussG with a summary of how I understand the solution and some potential effects [01:11:11] ejegg: cool thanks! [01:11:16] heh sorry we were just eating [01:11:36] ejegg: wrt the 1-hr tests, yes they'd probably have to last longer, but they probably should anyway [01:13:01] ejegg: just gonna put away some leftovers, then walky wog, then I'll be back more fully! [01:20:20] ok. i'm going to let these conversions run and i'll be back in a few hours to get frdb1003 back in service. [01:33:12] ejegg: also we could look at existing banner history data sets to figure out how many users might miss seeing banner because first pageview [01:52:14] yep yep [01:52:19] ok, dinner time here too [02:38:02] and... now it's late. gonna pack up a bit and head to bed. cya folks! [02:46:00] hmm, https://arxiv.org/pdf/1812.00474.pdf says session_length ≥ 3 is only true for 20-25% of users [02:51:13] ejegg|away: hey interesting! [02:51:27] thanks for writing all that up btw! [02:51:32] just adding a few more notes there :) [04:11:03] (PS1) Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/670999 [04:12:01] (CR) Eileen: [C: +2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/670999 (owner: Eileen) [04:12:39] (Merged) jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/670999 (owner: Eileen) [04:14:46] !log tools revision changed from d64b2f8cee to 532f8ecb33 [04:14:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:19:58] (PS3) Eileen: Add tools repo to checkout [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/670612 (https://phabricator.wikimedia.org/T269708) [04:20:41] (CR) Eileen: "> Patch Set 2:" (1 comment) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/670612 (https://phabricator.wikimedia.org/T269708) (owner: Eileen) [05:40:20] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, I18n: Months-long delays in deployment of CentralNotice translations - https://phabricator.wikimedia.org/T273176 (AndyRussG) >>! In T273176#6877506, @Pikne wrote: > These (lacking) translations are much more prominent now that there's "Banners"... [15:43:51] Fundraising-Backlog: Report request - https://phabricator.wikimedia.org/T277305 (SHust) [15:57:37] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 152181 Threads: 19 Questions: 1945755 Slow queries: 155 Opens: 11423475 Flush tables: 1 Open tables: 200 Queries per second avg: 12.785 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:02:20] hi fr-tech :) [16:08:59] Fundraising-Backlog: Remove the endowment trigger - https://phabricator.wikimedia.org/T277307 (SHust) [16:12:31] howdy AndyRussG [16:12:44] dwisehaupt :) [16:13:16] update on the dbs. hit a replication snag that delayed frdb1003 and frdev1001. that has now been cleared. [16:14:05] we should be good to test the silverpop code against the converted db in the next 30-45 minutes on our side. [16:14:17] ah cool! [16:14:21] yay! :) [16:23:59] Fundraising-Backlog, FR-Civi-Dedupe: Report request - https://phabricator.wikimedia.org/T277305 (DStrine) [16:24:17] Fundraising-Backlog, FR-Civi-Dedupe: Remove the endowment trigger - https://phabricator.wikimedia.org/T277307 (DStrine) [16:50:56] Fundraising-Backlog, FR-Civi-Dedupe: Dedupe hover tool tip should show a total of all contributions - https://phabricator.wikimedia.org/T277307 (DStrine) [16:51:49] Fundraising-Backlog, FR-Civi-Dedupe: Dedupe hover tool tip should show a total of all contributions - https://phabricator.wikimedia.org/T277307 (DStrine) I've edited this for a little more clarity. I will have to check with other teams to see if this interferes with their use cases. [17:14:50] Fundraising-Backlog, FR-Civi-Dedupe: Dedupe hover tool tip should show a total of all contributions - https://phabricator.wikimedia.org/T277307 (LeanneS) Just adding that these are the the two fields we thought would help to distinguish when hovering if we need to be picky on number of fields: Endowment... [17:36:59] i looked at T276293 and it looks like the code change removing the last utf8 bits for silverpop was merged. [17:36:59] T276293: Test silverpop export with utf8mb4 tables - https://phabricator.wikimedia.org/T276293 [17:37:50] i think i just need some help verifying that it was deployed and then running a test of the silverpop_emails_export_only job and then the full export and upload jobs. [17:45:03] grabbing breakfast. back in a few. [17:53:55] dwisehaupt: it looks like it got deployed last night [18:01:20] cstone: cool. [18:04:41] ah. so i should be able to run a job by using `run-job --job $jobname` correct? [18:10:15] dwisehaupt: sorry laptop meltdown yeah and I dont think you need the --job [18:11:19] ok cool. i'll give it a bo. [18:11:23] go even. [18:16:18] ok it ran, unfortunately it auto deletes and file older than 1 day. i have file sizes for comparison but not content. [18:17:41] unsubscribes, optout, and matching_gifts match byte size. database update shrunk to 32214184 from 59812628 [18:17:47] checking to see what that means. [18:26:07] Wikimedia-Fundraising-Banners: QA for enIT Banners - https://phabricator.wikimedia.org/T275539 (jbolorinos-ctr) Screenshot Test Results: - Dsk Lg: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zee098c9d56d365507db - Dsk Sm: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb... [18:26:25] ok. getting a fun error on testing the db update not just the export: pymysql.err.InternalError: (1267, "Illegal mix of collations (utf8mb4_general_ci,IMPLICIT) and (utf8mb4_unicode_ci,IMPLICIT) for operation '='") [18:40:07] trying to hunt for that but phpstorm is refusing to search [18:41:34] ok. i think i see what happens. [18:41:56] when the tables get recreated, silverpop_countrylangs keeps getting created with a collation of utf8mb4_general_ci [18:42:15] this is despite the default collation for the silverpop db being utf8mb4_unicode_ci [18:43:39] utf8mb4_unicode_ci is also the default server collation, so i'm not quite sure where it is picking up that value to use. [18:44:07] yeah the countrylangs was included in the update hmm [18:44:15] oh isee tt [18:44:30] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/670000/1/silverpop_export/silverpop_countrylangs.sql [18:44:35] a general snuck in the top there [18:44:58] ah yes. [18:45:15] dunno why that didn't cause an issue under 3 byte utf8. [18:48:33] (PS1) Cstone: Switch general to unicode. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/671217 (https://phabricator.wikimedia.org/T276293) [18:50:43] (CR) jerkins-bot: [V: -1] Switch general to unicode. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/671217 (https://phabricator.wikimedia.org/T276293) (owner: Cstone) [18:51:31] hah its failing cause of the same error [18:52:22] ok. so that should dtrt if nothing else relies on the general collation (which i don't think it does. [18:52:48] otherwise we could force a collation on the select that has the join in update_silverpop_staging.sql [18:53:16] dwisehaupt: want to look over my change ^ above and I'll force it through since CI is broken because of it somehow [18:55:12] since the drupal contribution tracking is unicode collation i think your change is the best fit. [18:56:18] okie ill push it through and redeploy then you can try again? [18:57:20] (CR) Dwisehaupt: [C: +2] "I think this is ok as long as we don't have anything else that relies on the general collation. I haven't found anything that does." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/671217 (https://phabricator.wikimedia.org/T276293) (owner: Cstone) [18:57:52] sounds good. [18:58:03] (CR) Cstone: [V: +2] Switch general to unicode. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/671217 (https://phabricator.wikimedia.org/T276293) (owner: Cstone) [19:03:08] (PS1) Cstone: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/671218 [19:04:32] (CR) Cstone: [C: +2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/671218 (owner: Cstone) [19:06:24] (Merged) jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/671218 (owner: Cstone) [19:09:21] !log tools revision changed from 532f8ecb33 to b7b4060c30 [19:09:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:09:55] ok dwisehaupt that change is out [19:10:22] cool. testing. [19:14:12] well, it hasn't failed yet. :) [19:22:51] Hey fr-tech I just connected to the channel but I can't see message history... We're deciding whether or not to postpone one of our email sends to Monday and just wanted to check on status for the exports today? [19:23:44] hi katers we are testing it now [19:29:49] Wikimedia-Fundraising-Banners: QA for itIT Banners - https://phabricator.wikimedia.org/T275538 (jbolorinos-ctr) **Screenshot Test Results** - **Desktop large** - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zcd18c1de64dee0bf277 - Last best: https://app.crossbrowsert... [19:40:23] fundraising-tech-ops: FY2020-21 Q3 maintenance window - https://phabricator.wikimedia.org/T268056 (Jgreen) [19:41:30] fundraising-tech-ops: Investigate decommissioning two eqiad-frack vlans - https://phabricator.wikimedia.org/T174203 (Jgreen) Stalled→Open [19:42:55] fundraising-tech-ops: FY2020-21 Q3 maintenance window - https://phabricator.wikimedia.org/T268056 (Jgreen) [19:42:57] Fundraising Sprint Esperantoland, Fundraising-Backlog, fundraising-tech-ops: Test silverpop export with utf8mb4 tables - https://phabricator.wikimedia.org/T276293 (Jgreen) [19:45:01] (CR) DannyS712: "This change is ready for review." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/671227 (owner: DannyS712) [19:45:10] (CR) DannyS712: [C: +1] Remove php entry point [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/669152 (https://phabricator.wikimedia.org/T140850) (owner: Umherirrender) [19:48:41] Fundraising Sprint Esperantoland, Fundraising-Backlog: Update exchange rates before Latin America Pre-test - https://phabricator.wikimedia.org/T275812 (Imacchiavello) Hey everyone if this task is still open to contribute i would love to join. [19:54:55] fundraising-tech-ops: FY2020-21 Q3 maintenance window - https://phabricator.wikimedia.org/T268056 (Jgreen) [19:57:33] Fundraising Sprint Esperantoland, Fundraising-Backlog: Update exchange rates before Latin America Pre-test - https://phabricator.wikimedia.org/T275812 (DStrine) Open→Resolved @Imacchiavello hi! This task is actually closed at this point. This is specific to the fundraising tech stack and only the... [20:07:27] (CR) Umherirrender: [C: +2] README: remove outdated info about supported MW version [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/671227 (owner: DannyS712) [20:11:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Find Matches Using Fishing Net is hanging - https://phabricator.wikimedia.org/T277338 (MBeat33) [20:20:04] Fundraising Sprint Esperantoland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Find Matches Using Fishing Net is hanging - https://phabricator.wikimedia.org/T277338 (DStrine) [20:20:52] Fundraising Sprint Esperantoland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Find Matches Using Fishing Net is hanging - https://phabricator.wikimedia.org/T277338 (DStrine) I pulled this into the sprint but there aren't that many people on right now. @Eileenmcnaug... [20:26:38] (Merged) jenkins-bot: README: remove outdated info about supported MW version [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/671227 (owner: DannyS712) [21:00:39] silverpop export is still chugging along but looking good so far. [21:03:57] nice dwisehaupt [21:14:26] thanks everyone! [21:19:52] Awesome [21:20:15] yay! [21:22:37] cstone hey so quick reminder/nag about pls sending any progress u have on the Guzzle task before u sign off today, if possible and if you feel it makes sense to do so? thanks and apologies for the bother! ;) [21:48:10] ok. it just finished. [21:49:32] ok. i'm going to fire off the upload job now. [21:53:30] and it looks successful.