[02:18:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Ingenico late October refunds not picked up from audit file - https://phabricator.wikimedia.org/T267622 (MBeat33) [02:21:27] (PS3) Eileen: Load right thank you default for stock [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640269 (https://phabricator.wikimedia.org/T259173) [02:23:24] (CR) Eileen: "jgleeson - it seems the php doesn't have the contact id at at stage - but I moved the code around in the js & I think it makes sense now &" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640269 (https://phabricator.wikimedia.org/T259173) (owner: Eileen) [07:45:21] (CR) jerkins-bot: [V: -1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/640306 (owner: L10n-bot) [08:14:24] (CR) Raimond Spekking: [C: +2] "false positive" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/640306 (owner: L10n-bot) [13:24:27] fundraising-tech-ops: Upgrade fundraising bastion servers to buster - https://phabricator.wikimedia.org/T266287 (Jgreen) [13:24:29] fundraising-tech-ops: Upgrade fundraising bastion servers to buster - https://phabricator.wikimedia.org/T266287 (Jgreen) Open→Resolved p:Triage→Medium a:Jgreen [13:24:31] fundraising-tech-ops: Epic: Upgrade fundraising servers to buster - https://phabricator.wikimedia.org/T254198 (Jgreen) [13:28:55] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Resolved→Open Reopening this task. We have emails scheduled for today at 14:00 UTC. Could today... [13:35:30] hi @jgleeson you around? [13:36:40] We need a rerun of the civi export, same problem as before: https://phabricator.wikimedia.org/T265352#6616114 [13:36:58] emails for today have been unscheduled [15:00:54] hey AndyRussG! question for you. I'm only seeing 0.01 sample rates in `event_sanitized.centralnoticebannerhistory` for Oct. 1 onward. I was looking specifically for the France campaign to check out banner history for donatewiki clickers. Would you be able to look at this with me? [15:09:09] !log updated fundraising python tools from 087a596d3a to 7853f426ee [15:09:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:46:26] eyener: hi! sure [15:47:23] eyener: that sounds right btw. It's the only sample rate, I think? Except for logs sent for donor clicks, which don't have that field, I think? [15:50:07] AndyRussG I would have expected a NULL sample rate from Oct 1 - Nov. 1 for the France campaign though since some of the logs would have been associated with donor clicks - right? [15:50:36] there would also be some NULL sample rates for en6C campaigns during that time [15:50:55] eyener: yes, I mean, I'm not sure how it shows up there when that field is not present? I'm happy to dig in [15:51:10] How are you querying? [15:51:42] if you have the time, I'd appreciate the help! I have a sample I can send [15:51:51] It's also possible there's something wrong with the system that copies to event_sanitiized and the events that were not sampled are dropped [15:51:55] is here a good place? [15:52:32] eyener: happy to help! [15:52:56] thanks AndyRussG! [15:52:58] eyener: of course just be careful not to drop anything that may have donor data here in this public channel [16:06:43] Hey all - channel switching here (not sure which one to use) - We'd like to get emails out the door today and the export jobs keep failing. Is the issue different from last time? @jgleeson @ejegg @mepps [16:07:13] katers let me grab stuff to look at logs [16:07:50] thx :) [16:08:49] hi fr-tech & katers :) [16:08:58] :) [16:09:12] hi! [16:09:42] argh did the export fail again! [16:10:05] Yeah and Elliott kicked it off a couple times and those failed as well today [16:10:26] oh that's not good [16:11:23] I feel like we should pull this into our next sprint fr-tech to add more context to the errors we were seeing with the silverpop export the last few weeks https://phabricator.wikimedia.org/T266253 [16:13:03] jgleeson fr-tech i'm looking at the log now, happy to jump on a call to try to work through this [16:13:17] it looks like emails got delayed because of this, is that right katers? [16:13:28] and jgleeson that sounds like a good idea about that task [16:14:26] anything useful in the logs mepps ? [16:14:43] jgleeson mepps lmk if you want to chat and I'll hop on too [16:14:43] i'm seeing a "duplicate key" error jgleeson [16:14:56] oh shoot [16:15:00] duplicate email? [16:15:11] I think we've had before [16:15:17] Yup @mepps - emails were supposed to go out 2 hours ago [16:15:22] thanks katers [16:15:51] so we were seeing an 'email cannot be null' error before, and I deployed eileen's latest fix [16:16:20] that fix changed the 5th query in update_table to do an inner join on civicrm_email rather than a left join [16:16:38] interesting ejegg [16:16:42] ejegg: can you link the ticket/patch please [16:16:59] actually I'll check the gitlog [16:17:03] where do the export files go jgleeson ejegg? [16:17:19] mepps i think it's under /var/lib/silverpop [16:17:45] oh interesting, we got a lot further this run [16:17:52] okay ejegg i confirmed there's no new file from today (just in case the error was a red herring) [16:18:11] all the way to wuery 32 Move the data from the staging table into the persistent one [16:18:43] yeah ejegg--and those table already exists errors are ones we usually get [16:18:59] so i was trying to figure out which is the real error--which appears to be this duplicate key [16:19:40] ah jeez those '--Query OK' comments are a little confusing [16:19:47] i think that last query was NOT ok [16:19:53] 2020-11-10 15:48:41,958 ERROR pymysql.err.IntegrityError: (1062, "Duplicate entry '36431250' for key 'PRIMARY'") [16:20:35] yeah jgleeson--it also doesn't say which table :/--can you see the table name anywhere? [16:20:46] I guess it's specific to query 32 [16:20:50] whatever tbl that is hitting [16:20:58] so the persistent table has a primary key on something that comes from civicrm_email.id [16:21:13] jgleeson mepps it's the insert into silverpop_export [16:23:55] interesting ejegg that we-use email id and that it would be duplicated [16:24:07] also that that wouldn't be caught in the staging table [16:24:42] so let's look at the rows in those tables right now [16:25:28] cool ejegg i'm in the db now [16:25:34] i don't see a duplicate entry for that key in staging [16:25:40] but we could try the query [16:27:23] hmm so far didn't get two rows [16:27:51] you running this query mepps https://github.com/wikimedia/wikimedia-fundraising-tools/blob/9757643003d0962cd348815f1b1ef0d965be642b/silverpop_export/update_table.sql#L277-L276 [16:28:20] yeah jgleeson though i was trying just doing the select with the primary key from the error message [16:28:28] so as not to get all of it... [16:28:30] yeah that makes sense [16:29:19] also possibly unrelated but what is silverpop_update_world ejegg [16:29:38] funny name, huh mepps? [16:30:02] it's a new table that shows just which contacts have been updated in the last week [16:30:08] ah okay ejegg [16:30:19] i can confirm that the email that tripped this up only occurs once in that table [16:31:02] hmm, so was it edited recently? [16:31:16] hmm but [16:31:39] there was already a row in silverpop_export that had the same id as the id that tripped us up [16:31:48] do we not have ON duplicate handling? [16:32:00] ahh, looks like eileen got rid of that [16:32:06] ahhh ejegg [16:32:36] ah [16:32:46] staying alive.. [16:33:04] ejegg i don't see the commit where that was removed [16:33:21] i guess what should we do to get this working today--also should we jump on a call? [16:33:24] jgleeson ^^ [16:33:43] ahh right I guess we never had it on that insert [16:33:43] sure [16:33:54] because we used to always be inserting into an empty table [16:34:18] jgleeson ejegg i'll be in the retro for today [16:34:24] i mean right now [16:34:28] that could have been confusing [16:52:36] katers we're trying to re-run again now [16:53:35] awesome, thanks! [16:53:44] (PS1) Mepps: Add duplicate handling for insert into silverpop export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/640470 [16:58:16] uploading files now katers [16:59:34] ty :) [17:00:54] katers new files should be uploaded [17:02:41] thanks! checking on them now. I appreciate the ping :) [17:05:17] fr-tech I removed those lines from the endowment thank you, can anyone look real quick https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/640264 [17:08:24] Hey @mepps I'm not seeing the files on the FTP... Could you confirm the timecode for those? [17:08:57] fr-tech: mail flowing through the frmx's is looking good with no current throttling by any major domains. we're going to bump up the traffic to 30% of the total runs. [17:09:22] that will let it bake in a few hours before the banner test in a few hours. [17:09:46] that's great to hear dwisehaupt [17:09:48] katers 20201110165740.csv [17:10:00] looking cstone [17:11:06] (CR) Jgleeson: [C: +2] "Looks good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640264 (owner: Cstone) [17:11:15] thanks jgleeson ! [17:11:24] re-running the upload katers [17:11:44] i thought i'd run it before... [17:12:37] Sorry, thanks for kicking it off again! For some reason Acoustic isn't registering any files on the FTP [17:12:47] okay can you look again now katers? [17:13:01] yes I'll re-log in too to be extra sure [17:13:03] !log upping thank you mail flow through frmx's to 30% of the total runs [17:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:13:26] wow just learned my thinkpad and charger and my mac charger are interchangable [17:13:28] by accident [17:13:50] amazing times we live in [17:16:00] @mepps ok I see them there now - thanks! [17:16:06] katers yay! [17:16:10] and phew :) [17:16:11] will let you know if it runs successfully [17:16:19] *fingers crossed* [17:17:47] ah, @mepps so that file is correct but now there's something up - bad data is 202k [17:18:02] woah katers ejegg ^^ [17:18:21] oh dang [17:18:41] let's take a look at the csv [17:18:59] I saw a preview of the file and it looks like the all_funds_first_donation_date format is changed to look like this: "b'12/16/2006" [17:19:06] I bet that's it [17:19:22] weird, didn't we fix that a while ago? [17:19:27] There's a random " b' " [17:19:48] hmm i see it in a bunch of places [17:21:20] i'm going to try to rebuild manually again [17:22:07] is that python code? that b' rings a bell with something i ran across with the dns to ipset code. related to utf8 iirc. [17:22:26] mepps ahh dwisehaupt has it [17:22:42] oh dwisehaupt ejegg? [17:22:52] what is the fix then and why would this have happened today? [17:22:55] err no [17:22:57] sorry [17:23:04] ? [17:23:26] i'm trying to rebuild the table again [17:23:28] we need to recreate those views again I think mepps [17:23:39] working on that now ejegg [17:28:25] @mepps also just heads up, i think the issue is only on DatabaseUpdate-* [17:28:37] thanks katers! [17:28:38] Seems like the unsubscribes file ran ok [17:31:29] its hard to be extra sure though, so feel free to send all over again if it's easier when we get there. Thank you all for working on this! [17:31:31] erg i still see that b! [17:32:58] mepps is it in the table? [17:32:59] ejegg the b's arent' in the views [17:33:05] damn damn [17:33:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Importing Activity Updates - https://phabricator.wikimedia.org/T219191 (jkim_wikimedia) Hi all, I wanted to check in on this task to see if the limit could be increased? Thanks! [17:33:25] (Merged) jenkins-bot: Update endowment thank you email text. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640264 (owner: Cstone) [17:33:32] dwisehaupt / Jeff_Green have there been some recent python package updates on civi1001? [17:33:56] looking [17:34:25] ejegg: there shouldn't have been anything beyond minor revs to the stock versions, what are you seeing? [17:34:55] Jeff_Green it's weirdly inserting a 'b' into the beginning of records in this csv [17:35:03] but when i look at the view, it's not there [17:35:07] it's outside the quotes too [17:35:16] so that would indicate it's not the selected value [17:35:16] that sounds like encoding fun [17:35:26] i guess it would be the csv package [17:35:28] Jeff_Green just the kind of fun we wanted! ;) [17:35:35] is this data coming out of mysql? [17:35:48] yeah, and being written with python csv package [17:36:30] I've been down this path, and I think I blocked what I saw from my memory :-P [17:36:45] any idea when it started? [17:36:45] there have been no python packages listed in the nightly upgrade report at least since oct 28th. [17:37:08] probably further back but i didn't scroll that far. i'll look at logs. [17:37:10] i think a mysql schema change can do it too [17:37:19] Jeff_Green: up till two nights ago it seems to have been exporting fine [17:37:23] ok [17:37:47] ejegg this is all i see about a b with the csv package: https://docs.python.org/2.6/library/csv.html#csv.writer [17:38:33] so that's a file io mode flag [17:38:38] i think meaning binary [17:38:48] ah, and might not apply for python3 [17:40:07] OH [17:40:11] it still applies ejegg [17:40:13] https://docs.python.org/3.0/library/csv.html#csv.writer [17:40:14] er nevermind [17:40:16] Jeff_Green [17:40:21] that was anticlimactic [17:40:29] I was thinking we had switched to frdb1004 as the reader, but we haven't [17:40:46] which would be a mysql version change from 10.1 to 10.4 [17:40:48] ah that would have been helpful Jeff_Green [17:40:51] but no [17:42:17] ejegg: this is happening with python3 right? [17:42:59] i don't see any apt or other package updates related to python on civi1001 [17:43:24] fwiw https://stackoverflow.com/questions/6269765/what-does-the-b-character-do-in-front-of-a-string-literal [17:44:02] I think when I've run into this in the past I've just done the .decode('UTF-8') hack mentioned there [17:45:19] is it possible the data itself i.e. what's been imported to mysql changed two days ago? [17:45:57] hmm what would i see in mysql Jeff_Green? [17:46:06] i'm trying to figure out where that decode statement would go [17:46:29] I'm not sure what you'd see, my guess is the mysql client would obscure it [17:47:16] ejegg Jeff_Green would it be here: https://github.com/wikimedia/wikimedia-fundraising-tools/blob/master/silverpop_export/export.py#L96 [17:47:34] no [17:47:36] mepps i.e. if variable foo prints that way you'd decode it first, i.e. foo_decoded = foo.decode('UTF-8) [17:48:12] did the type of that column get set to something funky? [17:48:27] i'm thinking somewhere here: https://github.com/wikimedia/wikimedia-fundraising-tools/blob/master/silverpop_export/export.py#L96 [17:48:47] could it maybe be a different client default encoding when you recreated the view? [17:48:55] it is a bunch of columns ejegg [17:49:02] hmm [17:49:15] I think it's also something you can control i.e. at connection time [17:49:34] looking at aiderator's db libarary I ended up with: [17:49:39] conf['db_conn'] = MySQLdb.connect( [17:49:45] charset='utf8', [17:49:52] use_unicode=True, [17:50:14] I'm not sure if this is relevant, it's been a long time, but I think it might be [17:51:05] that's ^^ python3, mariadb 10.4 [17:51:37] and I started running into encoding issues somewhere along the way because aide is collecting filenames [17:52:19] it looks like it is set somewhere [17:52:28] but maybe we do need to update the view? [17:53:10] yeah, somehow those columns are all varbinary [17:53:49] when they should be varchar, i think [17:54:39] mepps want me to try running the view update? [17:54:51] sure ejegg [17:55:06] i can run the first query but the set @sql part doesn't work for me [17:55:14] i'm surprised that's where this would have happened though [17:58:08] how's it going ejegg? [18:00:13] looks like they're varchars now [18:00:26] trying to figure out what would have been different when you did it [18:00:47] (I just copy-pasted the export_view_full command) [18:01:04] so let's re-run the file dump and upload jobs [18:01:36] running [18:07:14] ok, dates look good in the csv [18:07:17] running the upload [18:09:12] katers: ok, I *think* we just uploaded a good version of the files [18:09:39] phew ejegg [18:10:00] thank you i'll check now [18:17:23] mepps do you have any funky settings in a .my.cnf file on the machine you connected from to do that view change? [18:17:45] nope ejegg just checked [18:18:09] but i had trouble running the second half of the query with the CONCAT--could that have caused something? [18:18:24] i wouldn't think so [18:18:42] it was the underlying _full view that was set to varbinary [18:18:48] that's the first query [18:19:16] Just ran the jobs and checking them now - no more bad data :) [18:20:18] great katers ! [18:21:08] well i'm glad it's settled [18:21:19] i'm going to take a short walk so i can see sunshine [18:23:40] Thanks everyone for your help! We will start rescheduling emails now [18:32:51] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) This issue is now resolved. We had a bump in the road with manual exports failing a few times, and then... [18:33:13] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Open→Resolved [18:38:50] Fundraising-Backlog: Add extra logging to provide more info on silverpop export missing email failure - https://phabricator.wikimedia.org/T266253 (jgleeson) Open→Resolved a:jgleeson Closing this as it looks like @Eileenmcnaughton fixed it in this [[ https://gerrit.wikimedia.org/r/c/wikimedia/fund... [18:52:58] Fundraising-Backlog, fundraising-tech-ops: Create new job to send Civi thank you emails over smtp through frmx hosts - https://phabricator.wikimedia.org/T267259 (Dwisehaupt) jobs updated to handle 30% of job runs. Continuing with monitoring of logs. [18:54:27] jgleeson: is the stock email still a wait to deploy? [18:55:26] Wikimedia-Fundraising-Banners, Wikipedia-iOS-App-Backlog, Patch-For-Review, iOS-app-Bonefish-On-A-Balloon: 2020 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T266290 (Tsevener) Added 1 more patch for this one - the more recent versions of the iOS app only recognize a... [19:02:47] hey cstone I need to review a small patch from eileen on that and then it should be good to go out [19:03:27] do you want to do that today? otherwise I can just cherry pick these two ty email ones over [19:04:45] yeah I'll do that shortly, shouldn't take long. thanks for checking with me to remind me! [19:05:42] okie sounds good jgleeson [19:09:13] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, FR-Amazon, fr-donorservices, MW-1.36-notes (1.36.0-wmf.18; 2020-11-17): Amazon unresponsive donation form - https://phabricator.wikimedia.org/T267070 (MBeat33) Open→Resolved closing per David, "unintentional CSS issue" [19:24:17] thank you mail runs still looking good at the upped level. no current issues. [19:41:30] dwisehaupt: yay [19:43:59] yeah. it's good. low volume still but we'll get a big jump in 3 hours. :) [19:46:24] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Add SmashPig config for payments wiki on Docker - https://phabricator.wikimedia.org/T266089 (AndyRussG) [19:47:18] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Get queues working on docker - https://phabricator.wikimedia.org/T262977 (AndyRussG) Additional update: made queues persistent. Pa... [19:48:14] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, MW-1.36-notes (1.36.0-wmf.16; 2020-11-03), Patch-For-Review: Make payments phpunit tests pass under Docker - https://phabricator.wikimedia.org/T266088 (AndyRussG) [19:48:38] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: XDebug on payments on Docker - https://phabricator.wikimedia.org/T266090 (AndyRussG) [19:48:54] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Payments on Docker: issue with composer cache directory - https://phabricator.wikimedia.org/T266158 (AndyRussG) [19:54:07] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Set up text that goes next to payments forms on Docker - https://phabricator.wikimedia.org/T266092 (AndyRussG) [19:55:54] (CR) Jgleeson: [C: +2] "Tested and working for me. Nice work!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640269 (https://phabricator.wikimedia.org/T259173) (owner: Eileen) [19:56:42] ok cstone that is +2'd. once merged the stock stuff can go out. are you wanting to deploy before sprint? [19:56:46] sprint retro* [19:57:15] actually probably a bit tight, we could do afterwards tho [19:57:21] ok nice jgleeson do you need to run stuff once its deployed? [19:57:30] yeah i can get it set up at least [19:58:31] awesome [19:58:43] erm yeah I need to update the custom fields stuff on live [19:58:49] once it goes out [19:59:02] and make the new fields active [19:59:45] I might just check the new stuff on staging too to be extra sure [20:00:01] okie [20:00:41] Fundraising-Backlog: Docker dev setup: Review, update, organize and comment payments-LocalSettings.php - https://phabricator.wikimedia.org/T266840 (AndyRussG) More related commits, in addition to the above: - [[ https://gitlab.com/andyrussg/fundraising-dev/-/commit/23ccdc667083b2469f4d5cf08939538572448d3a |... [20:00:45] Fundraising-Backlog: Docker dev setup: Review, update, organize and comment payments-LocalSettings.php - https://phabricator.wikimedia.org/T266840 (AndyRussG) [20:02:18] Fundraising-Backlog: Docker dev setup: Set up memcached - https://phabricator.wikimedia.org/T267500 (AndyRussG) [20:06:04] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: fr-tech to look into supporting mail fail over for civi - https://phabricator.wikimedia.org/T264663 (DStrine) Open→Resolved [20:12:40] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Monthly convert pwk modal variant 011 (alt button text) - https://phabricator.wikimedia.org/T266562 (DStrine) Open→Resolved [20:12:56] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Monthly convert pwk modal variant 012 (alt button text and button order) - https://phabricator.wikimedia.org/T266563 (DStrine) Open→Resolved [20:13:18] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Monthly convert pwk modal variant 013 (alt button text and button order) - https://phabricator.wikimedia.org/T266564 (DStrine) Open→Resolved [20:13:20] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog: Monthly convert pwk modal variant 014 (alt button text and button order) - https://phabricator.wikimedia.org/T266590 (DStrine) Open→Resolved [20:13:50] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 3 others: Adyen visa debit users see error coming back from card processor - https://phabricator.wikimedia.org/T264941 (DStrine) Open→Resolved [20:14:27] Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: endowment donor did not get TY email. - https://phabricator.wikimedia.org/T253257 (DStrine) Open→Resolved [20:17:30] (Merged) jenkins-bot: Load right thank you default for stock [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640269 (https://phabricator.wikimedia.org/T259173) (owner: Eileen) [20:23:14] (PS1) Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640515 [20:27:16] (CR) Cstone: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640515 (owner: Cstone) [20:44:49] (CR) Cstone: [C: +2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640515 (owner: Cstone) [20:49:14] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Thank you letter not showing in the Activity field for Endowment Donation - https://phabricator.wikimedia.org/T239745 (Eileenmcnaughton) I think this duplicates https://phabricator.wikimedia.org/T253257 [21:00:12] Fundraising Sprint W 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Ingenico late October refunds not picked up from audit file - https://phabricator.wikimedia.org/T267622 (DStrine) [21:00:14] Fundraising Sprint W 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Test whether thank-you-mail slow query runs on RO-server - https://phabricator.wikimedia.org/T267604 (DStrine) [21:00:16] Fundraising Sprint W 2020, Fundraising-Backlog, fr-donorservices: Can we have a separate frCA TY email? - https://phabricator.wikimedia.org/T267250 (DStrine) [21:00:18] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, Patch-For-Review: Stripe Import error: INVALID_FILE_FORMAT This file is missing column headers - https://phabricator.wikimedia.org/T267255 (DStrine) [21:00:22] Fundraising Sprint W 2020, Fundraising-Backlog: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (DStrine) [21:00:24] Fundraising Sprint W 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, fr-donorservices: Civi: Recur fail message sent to active recurring donor - https://phabricator.wikimedia.org/T260910 (DStrine) [21:00:31] Fundraising Sprint W 2020, Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: Adyen message in queue without pending entry - https://phabricator.wikimedia.org/T259253 (DStrine) [21:00:33] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, and 2 others: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (DStrine) [21:00:35] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, FR-AutoTY-Email, and 2 others: Endowment TY email review Q2 2020 - https://phabricator.wikimedia.org/T267336 (DStrine) [21:00:37] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, FR-Email: Investigate email preference center prototype - https://phabricator.wikimedia.org/T266607 (DStrine) [21:00:39] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog: matching gifts values don't match between databases - https://phabricator.wikimedia.org/T266475 (DStrine) [21:00:41] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, Android-app-Bugs, and 6 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (DStrine) [21:00:46] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Civi: set Forget Me function to include display_name for RML emails - https://phabricator.wikimedia.org/T265652 (DStrine) [21:00:48] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, FR-Smashpig: Make cURL debug logging useful again - https://phabricator.wikimedia.org/T264931 (DStrine) [21:00:50] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog: Payments on Docker: issue with composer cache directory - https://phabricator.wikimedia.org/T266158 (DStrine) [21:00:52] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, and 3 others: Automatic TY for Stock Gifts - https://phabricator.wikimedia.org/T259173 (DStrine) [21:00:54] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog: Add SmashPig config for payments wiki on Docker - https://phabricator.wikimedia.org/T266089 (DStrine) [21:00:56] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog: XDebug on payments on Docker - https://phabricator.wikimedia.org/T266090 (DStrine) [21:00:58] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog: Set up text that goes next to payments forms on Docker - https://phabricator.wikimedia.org/T266092 (DStrine) [21:01:00] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising-Backlog, and 2 others: Make payments phpunit tests pass under Docker - https://phabricator.wikimedia.org/T266088 (DStrine) [21:01:02] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Get queues working on docker - https://phabricator.wikimedia.org/T262977 (DStrine) [21:01:04] Fundraising Sprint Octopus hugs, Fundraising Sprint Pseudopretzels, Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint Upbeat Upholstery, and 6 others: Fr-tech chores list - https://phabricator.wikimedia.org/T258527 (DStrine) [21:01:06] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint W 2020, Fundraising Sprint 🐍 is not a valid zipcode, and 5 others: Get payments wiki working on docker - https://phabricator.wikimedia.org/T262976 (DStrine) [21:28:08] (CR) Jgleeson: [C: +2] "this is good to go as discussed. nice!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/639448 (https://phabricator.wikimedia.org/T267255) (owner: Eileen) [21:28:43] !log civicrm revision changed from b1342c4129 to c36a5cc1b1 [21:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:31:18] dstrine: endowment ty is deployed [21:31:28] thanks cstone !!!! [21:34:19] (CR) jerkins-bot: [V: -1] Update Stripe file mapping [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/639448 (https://phabricator.wikimedia.org/T267255) (owner: Eileen) [21:45:02] fr-tech: the thank you send failmail seems to be related to missing the column s.stock_value on a query. not related to the smtp code for the frmx's. [21:45:44] jgleeson ^^ [21:48:04] dwisehaupt: argh- we are trying to fix [21:48:42] (PS1) Jgleeson: Remove deleted prior_wmf_giving custom field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640560 [21:48:43] the upgrade script tried to add a field that was deleted last year - but not from our fields list [21:49:00] cool. just hopped on it since i want to make sure the smtp process wasn't going awry. [21:49:15] (CR) Eileen: [C: +2] Remove deleted prior_wmf_giving custom field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640560 (owner: Jgleeson) [21:49:47] yeah - trying to redelete but now in alter hell [21:50:27] dwisehaupt: can you check trigger on civicrm_value_1_prospect_5 doesn't mention 'prior' [21:51:11] (CR) Eileen: [V: +2 C: +2] Remove deleted prior_wmf_giving custom field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/640560 (owner: Jgleeson) [21:56:09] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640561 [21:56:24] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640561 (owner: Eileen) [21:56:27] (CR) Eileen: [V: +2 C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/640561 (owner: Eileen) [21:56:48] i don't see it there. [21:57:04] triple checking [21:58:11] !log update civicrm revision changed from c36a5cc1b1 to d0cd7f6dbb [21:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:59:58] yeah. looks clear. [22:01:20] cool - I think we sorted the issue [22:01:21] hmmm... it may have horned up the replication on frdb1003. [22:01:29] oh dang [22:01:51] Error 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs' on query. Default database: 'civicrm'. Query: '/* User : 0 */ALTER TABLE civicrm_value_1_prospect_5 [22:02:01] ug [22:02:15] the thing is we added & removed the extra column :-( [22:03:04] yeah. looks like it may have been on the 'ADD COLUMN `prior_wmf_giving_234` tinyint'' [22:03:12] looking at options. [22:03:21] yep - it shouldn't have been added [22:04:16] ok. well, that makes it a little easier, since if it's just this one statement, we could skip it. before doing that. i want to check the other dbs. this and frdb1004 are the 2 10.4 hosts. [22:05:01] Fundraising-Backlog: alter civicrm.log_civicrm_value_1_stock_information_10 failure - https://phabricator.wikimedia.org/T267688 (jgleeson) [22:06:28] frdb1004 is ok. that's good. [22:09:33] Wikimedia-Fundraising-Banners, Wikipedia-iOS-App-Backlog, Patch-For-Review, iOS-app-Bonefish-On-A-Balloon: 2020 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T266290 (Tsevener) This is ready for QA now. This also opens up https://phabricator.wikimedia.org/T267064 to... [22:16:14] ok. i skipped the addition of that column. then restarted replication and skipped the statement that was the drop of that column [22:17:37] eileen: ok. not sure where this leaves us but replication is healthy again. [22:18:14] is there anything else you need from us to help make sure this doesn't recur? also a bit odd that this only happened on the one host. [22:23:38] thanks dwisehaupt [22:26:24] fwiw, here's what caught us up on this host. not sure why it just happened there, but doing some digging: https://mariadb.com/kb/en/troubleshooting-row-size-too-large-errors-with-innodb/ [22:27:46] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, Patch-For-Review: Stripe Import error: INVALID_FILE_FORMAT This file is missing column headers - https://phabricator.wikimedia.org/T267255 (Eileenmcnaughton) @jkim_wikimedia can you test now by j... [22:28:38] dwisehaupt: the issue I think is that the ALTER statement was trying to added a tinyint with a length of 255 [22:30:33] not very tiny... [22:31:02] lol [22:34:55] yeah. not very tiny at all. :) what triggered the alter? [22:35:25] sorry to correct that, a size of 255 not length and I think you can have a tinyint with a max size of 255 but it has to be unsigned [22:37:52] dwisehaupt: we added a new drush command to run the update_custom_fields routine which creates new custom fields for civicrm. When we ran it for the first time before it ran through all the previously applied custom fields (as we hoped) but it found a custom field entry which had been deleted at a later point via the web UI and not cascaded back to the list of custom fields in the code [22:38:23] added and then deleted* [22:40:34] so it meant the cli cmd compared the code with the live db and thought that tinyint field needed add in and when it tried to add it, the length constraint tripped [22:40:51] adding in* [22:42:00] at least that's what we /think/ happened [22:42:54] eileen: that email came through on its own [22:42:59] so that looks good [22:43:22] jgleeson: ah. ok cool. [22:43:28] have a good rest of your day and log weekend fr-tech. catch you next week [22:43:34] wasn't aware that alters could happen that way. [22:46:13] (CR) Eileen: "recheck" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/639448 (https://phabricator.wikimedia.org/T267255) (owner: Eileen) [22:55:53] fr-tech I'll be a few minutes late too [23:42:38] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, Patch-For-Review: Stripe Import error: INVALID_FILE_FORMAT This file is missing column headers - https://phabricator.wikimedia.org/T267255 (jkim_wikimedia) Hi @Eileenmcnaughton still showing the...