[01:01:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Thank you mail send failed - https://phabricator.wikimedia.org/T376893#10248809 (10AnnWF) it self resolved a bit later https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=53781380&selectedChild=activity, this error comes when we can not track t... [01:23:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Possible email search bug - https://phabricator.wikimedia.org/T374260#10248830 (10AnnWF) DB treat _(underscore) as wildcard for a single placeholder, and %(percent) is a wildcard character that can match zero or more characters in DB... [01:29:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: "Zendesk TIckets" tab not showing related tickets on for some donors - https://phabricator.wikimedia.org/T374154#10248835 (10AnnWF) Hi Jack, jgleeson. I am not able to see the Zendesk on prod, but as your previous patch our logic is to show zendesk ticket... [07:02:20] (03CR) 10Nikerabbit: [C:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082019 (owner: 10L10n-bot) [10:10:57] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10249440 (10cmooney) >>! In T377381#10246886, @Jgreen wrote: >> That's a bit of a shame in some ways but no... [13:17:11] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog, 07payments-orchestration: Set up production Gravy Adyen 3DSecure Scheme config - https://phabricator.wikimedia.org/T375214#10250134 (10jgleeson) Last night, we had a one-hour test covering 6ENC countries with 3D Secure enabled. Unfortunately, w... [13:17:51] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog, 07payments-orchestration: Set up production Gravy Adyen 3DSecure Scheme config - https://phabricator.wikimedia.org/T375214#10250136 (10jgleeson) 05Resolved→03Open [13:24:10] Random observation: a CI job quibble-fundraising-cldr-REL1_39-php74 exists but should be deprecated? It seems that donate.wmo is deployed with MW 1.43 along with the train now? [13:25:44] hi awight ! We still have cldr as a requirement for DonationInterface as well [13:25:55] and that's on 1.39 till January [13:26:08] aah TY :-) [13:26:53] I'm just being lazy so no worries... I made a few patches to cldr and the fundraising CI job fails, so my instinct was to blame the job rather than my patch ;-) [13:34:30] hehe [13:42:44] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1082210 [13:42:48] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1082210 (owner: 10Ejegg) [13:44:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail mail received - https://phabricator.wikimedia.org/T377845#10250262 (10jgleeson) [13:50:37] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog: Add address fields for all Gravy 3D Secure countries - https://phabricator.wikimedia.org/T377846 (10jgleeson) 03NEW [13:51:08] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1082210 (owner: 10Ejegg) [13:56:59] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: CIDs with PayPal cancel date of 1969-12-31 need canceling on Paypals end - https://phabricator.wikimedia.org/T373947#10250301 (10Ejegg) Hi @AMJohnson. I just made and cancelled a recurring of my own and I did indeed get an email: | W... [14:09:44] !log payments-wiki upgraded from 7ae3479f to a039cd50 [14:09:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:13:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 4993 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:13:12] 84% inode=99%): /srv 482407 MB (10% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [14:14:41] uh ^ [14:18:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 4993 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:18:12] 84% inode=99%): /srv 467398 MB (10% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [14:23:08] Jeff_Green: any idea what's up with those disk errors? [14:23:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 4993 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:23:12] 84% inode=99%): /srv 454994 MB (9% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [14:24:47] ejegg: it's a little puzzling actually, we're right around 80% for /srv which is right at the warn threshold, but I'm not clear why it's saying critical yet [14:28:07] running the check manually, I get a "warning" not "critical" so it shouldn't be paging [14:28:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 4993 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:28:12] 84% inode=99%): /srv 443705 MB (9% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [14:29:02] oh wait, I'm looking at civi1002 (which was also alerting earlier) [14:29:13] frdb1004 is surely the db dump [14:30:23] I think this is the leading edge of "too much data" :-) [14:30:50] eek [14:31:17] the issue here is that the nightly dump produces a 1.1T temp file before gzipping it down to a friendlier 1.1GB [14:32:39] we could try gzip on the fly, but historically we did not b/c of trying to get the database step done as quickly as possible [14:33:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 4993 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:33:12] 84% inode=99%): /srv 437728 MB (9% inode=99%): /run/user/571 51459 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [14:34:08] another option would be to separate log dumps from regular table dumps, but we'd have to lock the db to make sure we get everything at the same binlog position [14:35:21] the newer database machines have more disk space [14:35:33] frdb1004 is on the older end [14:36:23] this is a long way of saying it's not a surprise, and it's not mission critical, but we'll have to make some adjustments [14:38:46] ack [14:41:46] aha [14:43:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail mail received - https://phabricator.wikimedia.org/T377845#10250597 (10NNgu-WMF) Also, the Email team has only been getting confirmation emails for changes to the MASTER SUPPRESSION LIST, when in the past we've gotten all 4 Contact Source Job update e... [14:46:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-PayPal-ExpressCheckout, 10FR-Smashpig: Add PayPal IPN type for recurring_payment_outstanding_payment_failed - https://phabricator.wikimedia.org/T324510#10250612 (10Ejegg) 05Resolved→03Open [14:52:22] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10250655 (10Jgreen) There are 6 servers being replaced: {T369565} {T369947} {T369947} Plus 3 new servers:... [14:55:50] (03CR) 10Damilare Adedoyin: [C:03+2] Remove duplicated test [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082061 (owner: 10Jgleeson) [14:55:54] ejegg: regarding civi1002, a large chunk of /srv is consumed by /srv/org.wikimedia.civicrm-files - and within that I'm wondering about directory js -- it looks like civi is storing both an uncompressed and a .gz for every stored file. Are both of those needed? [14:56:48] getting rid of the uncompressed ones would free up about 10% of that partition [14:57:19] (03Merged) 10jenkins-bot: Remove duplicated test [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082061 (owner: 10Jgleeson) [14:58:00] (03PS1) 10Ejegg: Remap outstanding payment IPN [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082228 [14:58:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 5000 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [14:58:12] 84% inode=99%): /srv 499693 MB (10% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [15:01:10] (03CR) 10Damilare Adedoyin: [C:03+1] "Thanks for this jgleeson, made a suggestion inline to add the ReconciliationId to the PaymentDetailResponse decorated by the GravyGetLates" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082062 (https://phabricator.wikimedia.org/T374144) (owner: 10Jgleeson) [15:03:09] Jeff_Green: I'm not sure about that gzipped javascript. It's amazing that JS files could be taking up so much space! [15:03:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 5000 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [15:03:12] 84% inode=99%): /srv 493472 MB (10% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [15:03:21] I'll poke around and see if we can delete some [15:03:40] it's about 46GB on a 300GB partition [15:03:55] thx, do you want me to create a task for it? [15:05:35] (03CR) 10Damilare Adedoyin: [C:03+2] Remap outstanding payment IPN [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082228 (owner: 10Ejegg) [15:06:43] (03Merged) 10jenkins-bot: Remap outstanding payment IPN [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082228 (owner: 10Ejegg) [15:06:45] another big consumer is /srv/archive/civi/audit, I think we may already have a task for that somewhere [15:08:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 5000 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [15:08:12] 84% inode=99%): /srv 486716 MB (10% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [15:10:15] how is it that we have almost a TB of binlogs?! [15:13:12] PROBLEM - check_disk on frdb1004 is CRITICAL: DISK CRITICAL - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 5000 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot [15:13:12] 84% inode=99%): /srv 478524 MB (10% inode=99%): /run/user/571 51459 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [15:13:46] and that's just 10 days! [15:14:43] having a hard time finding any docs on the CiviCRM js directory [15:16:49] but they seem to be autogenerated [15:16:56] and go back to early september [15:17:05] ya [15:17:09] I'm guessing we can prune a lot closer to now [15:19:22] I don't think it's urgent enough to risk it, but deduping/reducing how it is use would be good [15:23:04] I'm sure eileen will have some more insight into it when she comes online [15:23:27] cool [15:27:52] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10250935 (10Jgreen) [15:36:17] Jeff_Green: I'm seeing problems for the civi server connecting to https://wikimedia.zendesk.com [15:36:25] do you think there's a network issue? [15:36:40] wenjun has a ticket here: https://phabricator.wikimedia.org/T377788 [15:36:52] looking [15:38:12] RECOVERY - check_disk on frdb1004 is OK: DISK OK - free space: /dev 257276 MB (100% inode=99%): /run 51458 MB (99% inode=99%): / 5000 MB (71% inode=89%): /dev/shm 257299 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /home 12348 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25122 MB (94% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23700 MB (89% inode=99%): /var/log/audit 422 MB (99% inode=99%): /boot 575 MB (84 [15:38:12] 9%): /srv 1710990 MB (37% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_disk [15:38:29] ejegg: looks like they changed their IPs [15:38:54] ahh [15:39:13] I think we should add this host to the dns_to_ipset automated policy update foo, I'll note on the task but it's going to take a bit for me to get to it [15:39:20] ok, thanks! [15:40:11] I can add the new IP to get you unblocked for now, but it will probably break again before long until we do the better fix [15:40:20] hi ejegg, just to confirm is donorwiki where we deploy emailpref? [15:41:02] ejegg: errr actually nm, I need to do a little research on those IPs first [15:43:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Failed to load zendesk - https://phabricator.wikimedia.org/T377788#10251045 (10Jgreen) They appear to have changed IPs for their service. Short term fix I will update our firewall policy to their new IPs. Longer term fix for which I'm keeping this task... [15:44:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10fundraising-tech-ops: Failed to load zendesk - https://phabricator.wikimedia.org/T377788#10251057 (10Jgreen) 05Open→03In progress p:05Triage→03Medium [15:46:09] damilare: yep, donorwiki [15:46:19] thanks [15:46:24] ejegg: zendesk IPs are updated, hopefully that unblocks it [15:46:33] thank you Jeff_Green ! [15:52:34] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy ACH (Trustly) - https://phabricator.wikimedia.org/T371810#10251092 (10Damilare) The Gravy-Trustly integration is blocked for the following reasons: - ~~Trustly yet to provide production credent... [15:53:23] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Map Processing status for Gravy/Trustly ACH to Smashpig Complete - https://phabricator.wikimedia.org/T377376#10251093 (10Damilare) a:03Damilare [16:18:18] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251197 (10Jgreen) [16:19:22] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251200 (10Jgreen) @IAckerman-WMF your SSH account and SSL client cert are set up, please see email for additional info. [16:26:36] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251248 (10Jgreen) [16:39:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: 0.00 native Google Pay payment - https://phabricator.wikimedia.org/T376195#10251274 (10AnnWF) a:03AnnWF [17:09:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: 0.00 native Google Pay payment - https://phabricator.wikimedia.org/T376195#10251449 (10AnnWF) currently the validateAmount is in web level only, we need to add additional check for this api call [17:13:44] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251462 (10Jgreen) [17:14:41] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251463 (10Jgreen) 05Open→03In progress p:05Triage→03High [17:17:04] 10fundraising-tech-ops, 10VPS-Projects, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10251491 (10taavi) Ping. [17:43:55] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251639 (10Jgreen) [17:44:29] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10251643 (10Jgreen) Database access is enabled including configured .my.cnf. [17:53:37] (03PS1) 10Wfan: Add amount validation for inapp payment submit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082255 (https://phabricator.wikimedia.org/T376195) [17:55:24] (03CR) 10CI reject: [V:04-1] Add amount validation for inapp payment submit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082255 (https://phabricator.wikimedia.org/T376195) (owner: 10Wfan) [17:56:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: CIDs with PayPal cancel date of 1969-12-31 need canceling on Paypals end - https://phabricator.wikimedia.org/T373947#10251672 (10SHust) @Ejegg Here’s a message that should help prevent confusion among donors, especially since some fr... [18:01:00] (03PS2) 10Wfan: Add amount validation for inapp payment submit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082255 (https://phabricator.wikimedia.org/T376195) [18:03:09] (03PS3) 10Wfan: Add amount validation for inapp payment submit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082255 (https://phabricator.wikimedia.org/T376195) [18:04:37] (03CR) 10Jgleeson: Capture Payment Orchestrator Reconciliation ID. (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082062 (https://phabricator.wikimedia.org/T374144) (owner: 10Jgleeson) [18:04:49] thanks damilare. I'll add that extra update [18:13:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail mail received - https://phabricator.wikimedia.org/T377845#10251743 (10AnnWF) related with T376950, this job's created_date is longer than 1 hour when job triggered, so we throw an exception "Out of date data_upload request found". [18:14:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: silverpop_emails_check_data_file_upload - Out of date data_upload request found - https://phabricator.wikimedia.org/T377845#10251745 (10AnnWF) [18:15:03] damilare: do you remember why you needed to add a default of a single space to the address fields on gravy requests? I feel like that was added to work around a validation bug on their end [18:15:28] yes that's exactly it [18:15:34] lemme find the slack thread [18:17:42] https://wikimedia.slack.com/archives/C070F1DVBRN/p1720529060421349 [19:01:05] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog: 2024 Sprint U - Priorities - https://phabricator.wikimedia.org/T376751#10251842 (10AKanji-WMF) [19:16:02] 06Fundraising-Backlog: FundraiseUp testing Q2 - 2425 - https://phabricator.wikimedia.org/T377732#10251894 (10AKanji-WMF) Flagging we need to have a conversation to determine data flows and where the source of truth will be @MSuijkerbuijk_WMF [19:19:10] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Get stats on number of donations resolved via Pending Transaction Resolver - https://phabricator.wikimedia.org/T374395#10251902 (10AKanji-WMF) Nice to have longer term [19:30:31] 06Fundraising-Backlog: Jazz up the mailcheck.js suggestion text - https://phabricator.wikimedia.org/T363473#10251946 (10AKanji-WMF) FYI we are pulling this in and are shipping the flames barring any concerns @spatton [19:53:57] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Add address fields for all Gravy 3D Secure countries - https://phabricator.wikimedia.org/T377846#10251992 (10AKanji-WMF) [19:53:58] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: FundraiseUp testing Q2 - 2425 - https://phabricator.wikimedia.org/T377732#10251994 (10AKanji-WMF) [19:54:00] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Map Processing status for Gravy/Trustly ACH to Smashpig Complete - https://phabricator.wikimedia.org/T377376#10251995 (10AKanji-WMF) [19:54:02] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review: Ensure Gravy one time tests only initiate one time donations - https://phabricator.wikimedia.org/T377335#10251996 (10AKanji-WMF) [19:54:08] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Implement exclusion for IPs generating artificially high levels of banner impressions - https://phabricator.wikimedia.org/T377268#10251998 (10AKanji-WMF) [19:54:09] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: 2024 Sprint U - Priorities - https://phabricator.wikimedia.org/T376751#10251999 (10AKanji-WMF) [19:54:16] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Acoustic SMS stage 5 - update queue consumer support for SMS RML - https://phabricator.wikimedia.org/T376686#10252001 (10AKanji-WMF) [19:54:17] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Acoustic SMS stage 4 - ensure the rml list in Acoustic criteria are updated - https://phabricator.wikimedia.org/T376692#10252000 (10AKanji-WMF) [19:54:18] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Check on outcome of upload job - https://phabricator.wikimedia.org/T376950#10251997 (10AKanji-WMF) [19:54:23] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Gravy Paypal Payments (recurring) - https://phabricator.wikimedia.org/T376470#10252004 (10AKanji-WMF) [19:54:27] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Figure out how translations of the email preference center will work - https://phabricator.wikimedia.org/T376514#10252002 (10AKanji-WMF) [19:54:31] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Set up recurring charge for venmo - https://phabricator.wikimedia.org/T376122#10252006 (10AKanji-WMF) [19:54:35] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Paypal - https://phabricator.wikimedia.org/T376469#10252005 (10AKanji-WMF) [19:54:39] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-email-preferences, 10MW-1.43-notes (1.43.0-wmf.28; 2024-10-22): Have a nicer error page when checksum is invalid - https://phabricator.wikimedia.org/T375798#10252007 (10AKanji-WMF) [19:54:43] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy Audit Edge Cases - https://phabricator.wikimedia.org/T376655#10252003 (10AKanji-WMF) [19:54:47] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#10252009 (10AKanji-WMF) [19:54:51] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Set up production Gravy Adyen 3DSecure Scheme config - https://phabricator.wikimedia.org/T375214#10252010 (10AKanji-WMF) [19:54:55] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10MW-1.43-notes (1.43.0-wmf.26; 2024-10-08), 13Patch-For-Review, 07payments-orchestration: Gravy Paypal Payments (one-time) - https://phabricator.wikimedia.org/T375378#10252008 (10AKanji-WMF) [19:54:59] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Gravy Google Pay - https://phabricator.wikimedia.org/T367806#10252013 (10AKanji-WMF) [19:55:03] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Gravy ApplePay - https://phabricator.wikimedia.org/T367785#10252014 (10AKanji-WMF) [19:55:07] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Scoping: 2nd Recurring Failure Email in Civi - https://phabricator.wikimedia.org/T365488#10252016 (10AKanji-WMF) [19:55:11] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Display Gravy reconciliation ID in CiviCRM - https://phabricator.wikimedia.org/T374144#10252011 (10AKanji-WMF) [19:55:15] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-Smashpig, 10FR-WMF-Audit, 13Patch-For-Review: PayPal audit processor and IPN listener should send payments to donations queue - https://phabricator.wikimedia.org/T366526#10252015 (10AKanji-WMF) [19:55:20] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy ACH (Trustly) - https://phabricator.wikimedia.org/T371810#10252012 (10AKanji-WMF) [19:55:24] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Jazz up the mailcheck.js suggestion text - https://phabricator.wikimedia.org/T363473#10252017 (10AKanji-WMF) [19:55:28] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-email-preferences: Email pref centre - Solve email overwrite issue from recurring - https://phabricator.wikimedia.org/T353376#10252020 (10AKanji-WMF) [19:55:32] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Add comms preferences footer to all Civi-generated emails - https://phabricator.wikimedia.org/T359445#10252019 (10AKanji-WMF) [19:55:36] 14Fundraising Sprint: justWork(), 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review: Add Auto Rescue for SEPA Direct Debit - https://phabricator.wikimedia.org/T362575#10252018 (10AKanji-WMF) [19:55:40] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-email-preferences, 10Recurring-Donations, 13Patch-For-Review: For donor prefs pages, offer to send a new link when checksum is expired - https://phabricator.wikimedia.org/T352337#10252021 (10AKanji-WMF) [19:55:44] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-donorservices: Civi "thank you letter summary" feature request - https://phabricator.wikimedia.org/T299096#10252022 (10AKanji-WMF) [19:55:50] 14Fundraising Sprint: didAnyoneTryThis(), 14Fundraising Sprint: justWork(), 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, and 5 others: Make unsubscribe work in EmailPreferences, delete Unsub extension - https://phabricator.wikimedia.org/T223330#10252023 (10AKanj... [19:55:54] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: [low priority] Investigate conditional Google's script addition for Google Ads Grant (fundraising) - https://phabricator.wikimedia.org/T374079#10252027 (10AKanji-WMF) [19:55:58] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 10FR-Tech-Analytics, 10FR-tech-data-integrity: Non Campaign Emails in Database - https://phabricator.wikimedia.org/T361621#10252026 (10AKanji-WMF) [19:56:02] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 13Patch-For-Review: Engage import_Matching - Ind contact import *updated - https://phabricator.wikimedia.org/T377605#10252025 (10AKanji-WMF) [19:59:47] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 10Wikimedia-Fundraising-CiviCRM: Batch Imports Tool - Scope Feature list - https://phabricator.wikimedia.org/T377890 (10AKanji-WMF) 03NEW [20:00:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10fundraising-tech-ops: Failed to load zendesk - https://phabricator.wikimedia.org/T377788#10252046 (10AnnWF) Confirmed the quick way now make the zendesk works again, moving to blocked due to "let's add this hostname to the dns_to_ipset dynamic updatin... [20:09:51] 14Fundraising Sprint: didAnyoneTryThis(), 14Fundraising Sprint: justWork(), 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, and 5 others: Make unsubscribe work in EmailPreferences, delete Unsub extension - https://phabricator.wikimedia.org/T223330#10252057 (10AnnWF... [21:10:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2036 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:15:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2279 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:20:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2528 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:25:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2761 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:30:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2993 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:35:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=3172 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:40:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=3407 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:45:24] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=3626 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:50:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=3889 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:55:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=4047 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [22:00:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=4219 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [22:05:24] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=4376 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [22:10:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=4522 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [22:11:21] ACKNOWLEDGEMENT - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=4522 [critical =2000] Jeff_Green known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [22:20:15] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10252289 (10Jclark-ctr) @cmooney Step 1: Firewall Installation & Cabling is complete Since we have racke... [22:21:01] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10252290 (10Jclark-ctr) [22:46:58] (03PS1) 10Ejegg: Add TY timing log lines [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082291 [22:47:44] (03CR) 10Eileen: [C:03+2] Add TY timing log lines [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082291 (owner: 10Ejegg) [22:54:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Get rid of some civi error when TY email - https://phabricator.wikimedia.org/T377895 (10AnnWF) 03NEW [22:56:22] (03CR) 10Ejegg: [V:03+2] Add TY timing log lines [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082291 (owner: 10Ejegg) [22:57:08] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082292 [22:57:13] (03PS1) 10Eileen: Decommission legacy function from legacy code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082293 [22:57:13] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082292 (owner: 10Ejegg) [22:57:34] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082292 (owner: 10Ejegg) [22:59:50] !log fundraising civicrm upgraded from 36660cb3 to d9e85c3d [22:59:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:12:11] (03PS1) 10Ejegg: More timing on TY mail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082294 [23:13:21] (03CR) 10Wfan: [C:03+2] More timing on TY mail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082294 (owner: 10Ejegg) [23:20:39] (03CR) 10Ejegg: [V:03+2] More timing on TY mail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082294 (owner: 10Ejegg) [23:22:03] (03PS1) 10Ejegg: More timing on TY mail [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082295 [23:22:03] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082296 [23:22:06] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082296 (owner: 10Ejegg) [23:22:25] (03Abandoned) 10Ejegg: More timing on TY mail [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082295 (owner: 10Ejegg) [23:22:34] (03Abandoned) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082296 (owner: 10Ejegg) [23:22:56] (03PS2) 10Eileen: Decommission legacy function from legacy code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082293 [23:22:56] (03PS1) 10Eileen: Yet more logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082297 [23:25:03] (03CR) 10Ejegg: [C:03+2] Yet more logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082297 (owner: 10Eileen) [23:25:14] (03CR) 10Ejegg: [V:03+2 C:03+2] Yet more logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082297 (owner: 10Eileen) [23:25:56] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082298 [23:25:59] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082298 (owner: 10Ejegg) [23:29:49] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082298 (owner: 10Ejegg) [23:32:37] !log fundraising civicrm upgraded from d9e85c3d to 5463f37b [23:32:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:33:45] (03PS1) 10Eileen: Migrate matching gifts notify email to a setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082299 [23:52:50] (03PS1) 10Eileen: temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082303 [23:53:43] (03CR) 10Ejegg: [C:03+2] temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082303 (owner: 10Eileen) [23:53:45] (03CR) 10Ejegg: [V:03+2 C:03+2] temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082303 (owner: 10Eileen) [23:54:45] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082305 [23:56:33] (03PS1) 10Eileen: more temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082306 [23:57:07] (03CR) 10Ejegg: [C:03+2] more temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082306 (owner: 10Eileen) [23:57:09] (03CR) 10Ejegg: [V:03+2 C:03+2] more temp logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082306 (owner: 10Eileen) [23:58:48] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082307 [23:58:50] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082307 (owner: 10Ejegg) [23:58:54] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082307 (owner: 10Ejegg)