[00:03:45] (PS2) Ejegg: Carry over original created date so the age of the entry is correct [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/641573 (https://phabricator.wikimedia.org/T222287) (owner: Eileen) [00:03:59] (CR) Ejegg: [C: +2] "Looks good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/641573 (https://phabricator.wikimedia.org/T222287) (owner: Eileen) [00:05:52] (CR) jerkins-bot: [V: -1] Carry over original created date so the age of the entry is correct [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/641573 (https://phabricator.wikimedia.org/T222287) (owner: Eileen) [00:05:56] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, and 3 others: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (bsisolak) I have opened a case up on these two jobs. [00:15:30] Wikimedia-Fundraising-Banners: [CANDIDATE] 2020-11-21 Pretest: Ask Strings - https://phabricator.wikimedia.org/T267983 (HNordeenWMF) [00:23:14] Wikimedia-Fundraising-Banners: [CANDIDATE] 2020-11-21 Pretest: Ask Strings - https://phabricator.wikimedia.org/T267983 (HNordeenWMF) @jbolorinos-ctr I just added the extra mobile large test! [00:28:31] (Merged) jenkins-bot: Carry over original created date so the age of the entry is correct [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/641573 (https://phabricator.wikimedia.org/T222287) (owner: Eileen) [01:21:28] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/641581 [01:21:46] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/641581 (owner: Eileen) [02:42:43] Fundraising Sprint We all meet again, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Categorize all the email export failures and suggest a solution - https://phabricator.wikimedia.org/T267757 (MNoorWMF) Thanks for looping, David! I think for data integrity's sake, what Eileen said... [02:48:08] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog: Stripe Import error: INVALID_FILE_FORMAT This file is missing column headers - https://phabricator.wikimedia.org/T267255 (Eileenmcnaughton) @jkim_wikimedia I just tried importing that file on live an... [07:27:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6199 5000, payments-init is 1626 1500, pending is 2001 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 12.91M (peak 12.95M, 0.21% of max, fragmentation 1.32%), connected_slaves is 2, donations is 1541, jobs is 0, jobs-adyen is 8, jobs-paypal is 715, recurring is 2, refund is 0, unsubscribe [07:32:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6223 5000, payments-init is 1633 1500, pending is 2014 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 12.92M (peak 13.02M, 0.21% of max, fragmentation 1.32%), connected_slaves is 2, donations is 1547, jobs is 0, jobs-adyen is 9, jobs-paypal is 716, recurring is 2, refund is 0, unsubscribe [07:37:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6263 5000, payments-init is 1642 1500, pending is 2030 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 12.99M (peak 13.08M, 0.21% of max, fragmentation 1.32%), connected_slaves is 2, donations is 1557, jobs is 0, jobs-adyen is 9, jobs-paypal is 722, recurring is 2, refund is 0, unsubscribe [07:42:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6304 5000, payments-init is 1654 1500, pending is 2050 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 13.08M (peak 13.20M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1568, jobs is 0, jobs-adyen is 10, jobs-paypal is 728, recurring is 2, refund is 0, unsubscrib [07:47:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6343 5000, payments-init is 1663 1500, pending is 2062 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 13.15M (peak 13.24M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1578, jobs is 0, jobs-adyen is 10, jobs-paypal is 734, recurring is 2, refund is 0, unsubscrib [07:52:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6384 5000, payments-init is 1672 1500, pending is 2075 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 13.25M (peak 13.30M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1587, jobs is 0, jobs-adyen is 10, jobs-paypal is 738, recurring is 2, refund is 0, unsubscrib [07:57:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6418 5000, payments-init is 1683 1500, pending is 2086 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 13.27M (peak 13.36M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1593, jobs is 0, jobs-adyen is 11, jobs-paypal is 742, recurring is 2, refund is 0, unsubscrib [08:02:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6437 5000, payments-init is 1686 1500, pending is 2096 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 10 hours - memory use is 13.31M (peak 13.41M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1596, jobs is 0, jobs-adyen is 11, jobs-paypal is 747, recurring is 2, refund is 0, unsubscrib [08:07:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6458 5000, payments-init is 1691 1500, pending is 2102 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 11 hours - memory use is 13.35M (peak 13.43M, 0.21% of max, fragmentation 1.31%), connected_slaves is 2, donations is 1601, jobs is 0, jobs-adyen is 11, jobs-paypal is 750, recurring is 2, refund is 0, unsubscrib [08:12:31] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6476 5000, payments-init is 1695 1500, pending is 2110 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 11 hours - memory use is 13.38M (peak 13.46M, 0.21% of max, fragmentation 1.30%), connected_slaves is 2, donations is 1604, jobs is 0, jobs-adyen is 12, jobs-paypal is 750, recurring is 2, refund is 0, unsubscrib [08:17:26] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6476 5000, payments-init is 1695 1500, pending is 2110 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 1 days 11 hours - memory use is 13.38M (peak 13.46M, 0.21% of max, fragmentation 1.30%), connected_slaves is 2, donations is 1604, jobs is 0, jobs-adyen is 12, jobs-paypal is 750, recurring is 2, refund is 0, un [08:17:26] sehaupt Queues backed up but flowing. [09:36:52] Fundraising-Backlog: Import job Failed - today Nov 18th - https://phabricator.wikimedia.org/T268120 (MSuijkerbuijk_WMF) [09:41:58] hi everybody! [09:42:06] Luca from Analytics :) [09:43:21] My team is planning a big upgrade to our Hadoop sw stack, that should have been completed in late November but we have to postpone for some issues on the hardware front [09:45:16] I am wondering if the upgrade can be scheduled in December as well, I recall last year that the Kerberos upgrade might have impacted some people using hadoop during a delicate time of the year for your team [09:45:49] in the kerberos case, client interfaces were changing a lot, but in this case nothing changes for the users, just more up to date tools [09:46:12] would it be ok from your side if Analytics schedules the upgrade in Dec? [09:46:21] lemme know, thanks :) [09:47:04] (the expected impact is a couple of hours of downtime) [12:19:15] Fundraising-Backlog: Import job Failed - today Nov 18th - https://phabricator.wikimedia.org/T268120 (jgleeson) Hi @MSuijkerbuijk_WMF thanks for logging this. I can confirm the export job hasn't run this morning as expected. I'm trying to work out why as there's nothing obvious which would explain it at the... [14:30:09] Fundraising-Backlog: Import job Failed - today Nov 18th - https://phabricator.wikimedia.org/T268120 (jgleeson) The job kicked off earlier completed successfully. However, I'm still not sure why the cronjob failed to run. [14:37:39] Wikimedia-Fundraising-Banners: November 19 post-payment monthly convert test - https://phabricator.wikimedia.org/T268045 (Pcoombe) @spatton All looks good to me! [14:42:33] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 1 days 17 hours - memory use is 10.48M (peak 24.50M, 0.18% of max, fragmentation 1.43%), connected_slaves is 2, donations is 2968, jobs is 0, jobs-adyen is 1, jobs-paypal is 3738, payments-antifraud is 0, payments-init is 5, pending is 0, recurring is 29, refund is 0, unsubscribe is 17 [16:09:15] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog: Stripe Import error: INVALID_FILE_FORMAT This file is missing column headers - https://phabricator.wikimedia.org/T267255 (jkim_wikimedia) @Eileenmcnaughton that's so strange! Thank you so much for yo... [16:45:39] Fundraising Sprint We all meet again, Fundraising-Backlog: Import job Failed - today Nov 18th - https://phabricator.wikimedia.org/T268120 (DStrine) [16:46:24] Wikimedia-Fundraising-Banners: [BETA] QA of TY Banners 2021 - https://phabricator.wikimedia.org/T268153 (jbolorinos-ctr) [16:47:05] Wikimedia-Fundraising-Banners: [BETA] QA of TY Banners 2021 - https://phabricator.wikimedia.org/T268153 (jbolorinos-ctr) Screenshot Test Results: - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z74de1bafaf54d550d20 - Mobile: https://app.crossbrowsertesting.com/public/i15b7... [16:51:58] Fundraising Sprint We all meet again, Fundraising-Backlog: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (Ejegg) @MDemosWMF does the comment above clear things up? [16:52:09] Fundraising Sprint We all meet again, Fundraising-Backlog: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (Ejegg) a:Ejegg [17:09:35] anybody reading? If so I posted a question earlier on, lemme know your thoughts when you have a moment :) [17:12:18] Fundraising Sprint We all meet again, Fundraising-Backlog: Add 'Stock Information' field to thank you letter - https://phabricator.wikimedia.org/T268081 (DStrine) [17:13:01] Fundraising Sprint We all meet again, Fundraising-Backlog: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (MDemosWMF) @Ejegg the financial type 'cash' is correct. Originally the gift was entered with payment method 'cash' as well in error, but when I wen... [17:21:08] hi elukey we're discussing in our daily standup [17:21:28] it sounds like it might be worth more discussion [17:21:39] elukey: is there a phab task about this or other documentation? Severl hours of downtime sounds like a major disruption for us [17:22:15] mepps: thanks! [17:22:54] sorry for the delay in reply, if you say fr-tech on this channel it will highlight it for everyone on fundraising tech [17:23:31] nono my bad, thanks for replying :) [17:23:53] dstrine: hi! we have https://phabricator.wikimedia.org/T255142 for the moment, but no date set in stone yet, this is why I am reaching out to your team [17:24:29] from our point of view it is not clear what is the usage of hadoop (and related tools) that fundraising does [17:24:41] so I am reaching out beforehand to have a chat [17:25:23] the downtime should be a couple of hours in theory [17:26:22] elukey: some of the advancement users don't engage with phab that much so I'm starting an email thread. Who else on your team would be good to loop in? [17:27:22] dstrine: I think me is fine for the moment, I'll add more people in if needed [17:27:37] ok thanks [17:31:49] AndyRussG: I gonna grab dinner now but we could chat about the stock info ticket when I get back if that works for you. back soon [17:43:17] Fundraising Sprint We all meet again, Fundraising-Backlog: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (MDemosWMF) @Ejegg I will also be at the civi fortnightly today, happy to discuss this then! [18:02:46] fundraising-tech-ops: Cron syntax linter for frdeploy tools and/or process-control - https://phabricator.wikimedia.org/T268158 (Jgreen) [18:03:47] fundraising-tech-ops: Cron syntax linter for frdeploy tools and/or process-control - https://phabricator.wikimedia.org/T268158 (Jgreen) Because we love regexes . . . https://stackoverflow.com/questions/14203122/create-a-regular-expression-for-cron-statement/57639657 :-) [18:15:36] Fundraising-Backlog, FR-AutoTY-Email: Copy request for recurring receipt and standard TY email - https://phabricator.wikimedia.org/T251428 (CDenes_WMF) i think we can resolve this. The monthly receipt was scrapped, and I have the TY email copy on hand! [18:16:14] Fundraising-Backlog, FR-Adyen: Too much whitespace between form and error messages [Adyen iframe] - https://phabricator.wikimedia.org/T264785 (EWilfong_WMF) Noting the CSS elements here that are setting the height on this iframe: * iframe itself has an inline style height of 500px; * #pmmdetails-card i... [18:17:30] Fundraising-Backlog, FR-AutoTY-Email: JAN Summary Emails 2021 need a signature - https://phabricator.wikimedia.org/T251424 (CDenes_WMF) Hi @DStrine! Checking up on this as we are now closer to the January send, and the emails definitely need the missing signature added at the end (for all languages). @MB... [18:23:37] Fundraising-Backlog, FR-AutoTY-Email: Copy request for recurring receipt and standard TY email - https://phabricator.wikimedia.org/T251428 (DStrine) Has it been scrapped? I'll need to follow up with others. I'll keep this open until then. [18:24:58] Fundraising-Backlog, FR-AutoTY-Email: JAN Summary Emails 2021 need a signature - https://phabricator.wikimedia.org/T251424 (DStrine) I'm hoping to get to this and other related tasks in the next few weeks. [18:36:17] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, and 3 others: Automatic TY for Stock Gifts - https://phabricator.wikimedia.org/T259173 (LeanneS) Much appreciated @jgleeson! [18:57:43] Fundraising Sprint We all meet again, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Request to create new $10k+ email bot - https://phabricator.wikimedia.org/T268066 (mepps) @CaitVirtue There is an error at the link you posted about but at https://civicrm.wikimedia.org/admin/config/large_donati... [19:00:57] Fundraising-Backlog, FR-AutoTY-Email: Copy request for recurring receipt and standard TY email - https://phabricator.wikimedia.org/T251428 (CDenes_WMF) That's what was communicated to me, but always prudent to get additional confirmation. [19:35:54] fr-tech just on why it was failing for me locally even after fixing stuff, turns out the broken result was cached [20:09:48] Fundraising Sprint We all meet again, Fundraising-Backlog: Import job Failed - today Nov 18th - https://phabricator.wikimedia.org/T268120 (Eileenmcnaughton) Open→Resolved a:Eileenmcnaughton @jgreen has since pinned this down to a typo in a new process-control job I created - which was messin... [20:28:13] Fundraising Sprint We all meet again, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Request to create new $10k+ email bot - https://phabricator.wikimedia.org/T268066 (mepps) Since we were able to create the alert and @RLewis sent me Shenya's email, I'm moving this to review. The bug is annoying... [20:56:54] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, and 3 others: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (Eileenmcnaughton) @MBeat33 @DStrine from what I can t... [20:59:09] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, and 3 others: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (DStrine) Thanks for this @Eileenmcnaughton sounds lik... [21:09:22] Fundraising-Backlog, Wikimedia-Fundraising, Wikimedia-Fundraising-Banners, Thank-You-Page, fr-donorservices: Following link to donate fails to return user to previous page upon completion - https://phabricator.wikimedia.org/T60506 (Pcoombe) Thanks @AndyRussG! Yes I did look at LocalStorage fi... [22:05:22] Wikimedia-Fundraising-Banners, Wikipedia-Android-App-Backlog (Android-app-release-v2.7.33x-R-Rosgulla): 2020 English fundraising messages in Android app - https://phabricator.wikimedia.org/T266289 (schoenbaechler) Tried this several times @cooltey. Can’t see the Fundraising message, see video below. htt... [22:12:28] Fundraising Sprint We all meet again, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Request to create new $10k+ email bot - https://phabricator.wikimedia.org/T268066 (CaitVirtue) Thanks all! [22:26:23] Wikimedia-Fundraising-Banners, Wikipedia-Android-App-Backlog (Android-app-release-v2.7.33x-R-Rosgulla): 2020 English fundraising messages in Android app - https://phabricator.wikimedia.org/T266289 (schoenbaechler) Looks good @cooltey! **01) Could you please use the dialog button set, **only for the dia... [22:46:46] Fundraising Sprint We all meet again, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Test whether thank-you-mail slow query runs on RO-server - https://phabricator.wikimedia.org/T267604 (Eileenmcnaughton) I guess we now know it doesn't @Ejegg I took a quick look at switching it over but there... [22:50:27] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, FR-Email: Investigate email preference center prototype - https://phabricator.wikimedia.org/T266607 (Eileenmcnaughton) So I'm gonna have a go at adding another docker image (just to my docker-com... [22:51:02] ejegg: AndyRussG just looking at https://phabricator.wikimedia.org/T266607 & testing locally & trying to decide if we would want 2 new images/containers to reflect 2 new servers (civiproxy & some external form server that connects to it) [22:53:18] oh hmm, eileen i think for that one we can use the payments machine to be the form server [22:53:28] since it will be built into donationinterface [22:53:35] ok so it would be 1 more then [22:53:44] and the only real difference would be config [22:54:00] oh 2 more containers, 1 more image? [22:54:08] i guess we don't want civiproxy running in the same image as civi? [22:54:28] ah no, I don't think we even need separate config in dev [22:54:40] well I think they are separate servers in real life? [22:54:40] so just the extra container/image for the proxy [22:54:58] ie where will civiproxy run - a port on civi1001 [22:55:05] or a separate in-between box? [22:56:04] I guess the test locally will be to call http://civiproxy:8000 & get the results & only the results we want [22:57:57] yeah, i think a port on civi [22:58:00] my understanding is that we'd run the civiproxy on the civi host. [22:58:24] on a separate port and then limiting contact to that port specifically. [23:01:49] ok cool - so perhaps not it's own container - just a change to the civi one [23:26:12] fundraising-tech-ops: Cron syntax linter for frdeploy tools and/or process-control - https://phabricator.wikimedia.org/T268158 (Dwisehaupt) I wrapped up a bunch of linting in the process-control job area today. With that, I see two places where we could do this. One would be writing one to run on the fundra... [23:29:22] Fundraising-Backlog, fundraising-tech-ops: Create new job to send Civi thank you emails over smtp through frmx hosts - https://phabricator.wikimedia.org/T267259 (Dwisehaupt) Hit a new error case to track today. We had a domain we were checking shift to not providing rdns entries. In this case, we were ge... [23:36:54] eileen: i'm looking at T268084, we could do this within puppet pretty easily. you mention "local dev details to be replaced with prod details" is this something that would need to be coded in as a template or more something that would be handled by devs on their local instances? [23:36:55] T268084: Create drush site aliases file - https://phabricator.wikimedia.org/T268084 [23:37:50] & I can dwisehaupt I've already added it to the WIP config for local devs - so via puppet on prod & staging would be great! [23:38:06] ah. ok. cool [23:38:46] a lot of that commentary is just explaining why wmff makes sense as an alias [23:38:53] i think to do this i'd probably just need the uri settings for each as i'm assuming they would change. [23:39:06] i can start templating it out. [23:39:22] right the uri settings are the url to reach the web interface [23:39:28] it is not that important [23:39:42] ie. the path is used for lots of things [23:39:56] the uri is used for a marginal number of things [23:40:18] so it could be excluded if it's a pain [23:40:29] cool. i'll give it a test in the vb setup to see how we would want to template it. [23:40:32] I'm pretty sure our process control jobs set -r but not the other [23:41:07] yeah so it's only prod & staging so the path is the same on both of them [23:41:33] in the docker image I've got the file being copied in in docker-compose [23:45:10] cool cool. [23:45:37] i think we have hooks already for "production" and "staging" in puppet, so it'll just be a matter to pulling in the right bits. [23:45:47] something fun to distract me at the end of the day. [23:47:07] :-)