[00:01:08] !log fundraising civicrm upgraded from 5463f37b to de642bea [00:01:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:14:19] (03PS1) 10Eileen: Revert "temp logging" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082309 [00:14:43] (03CR) 10Eileen: [C:03+2] Revert "temp logging" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082309 (owner: 10Eileen) [00:43:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: target_contact_id is not valid : 61146956 - https://phabricator.wikimedia.org/T377903 (10AnnWF) 03NEW [00:46:16] (03Merged) 10jenkins-bot: Revert "temp logging" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082309 (owner: 10Eileen) [00:48:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: silverpop_emails_check_data_file_upload - Out of date data_upload request found - https://phabricator.wikimedia.org/T377845#10252539 (10Eileenmcnaughton) I think those messages you got were during our testing & hopefully you won't be seeing them recur unl... [00:54:40] (03PS1) 10Eileen: Latest upstream 5.79rc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082311 [00:56:44] (03PS1) 10Eileen: Temporary fix for handling duplicate contacts on import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082313 (https://phabricator.wikimedia.org/T374063) [00:56:46] (03PS1) 10Eileen: Fix import to handle creating new donor advised organizations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082314 (https://phabricator.wikimedia.org/T375039) [00:56:48] (03PS1) 10Eileen: Fix dedupe query speed - port https://github.com/civicrm/civicrm-core/pull/30591 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082315 (https://phabricator.wikimedia.org/T353971) [00:57:01] (03CR) 10CI reject: [V:04-1] Latest upstream 5.79rc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082311 (owner: 10Eileen) [00:59:28] (03CR) 10CI reject: [V:04-1] Fix dedupe query speed - port https://github.com/civicrm/civicrm-core/pull/30591 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082315 (https://phabricator.wikimedia.org/T353971) (owner: 10Eileen) [00:59:30] (03CR) 10CI reject: [V:04-1] Fix import to handle creating new donor advised organizations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082314 (https://phabricator.wikimedia.org/T375039) (owner: 10Eileen) [01:00:05] (03CR) 10CI reject: [V:04-1] Temporary fix for handling duplicate contacts on import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082313 (https://phabricator.wikimedia.org/T374063) (owner: 10Eileen) [01:00:38] 06Fundraising-Backlog, 10FR-donorservices: Civi and acoustic odd behavior - https://phabricator.wikimedia.org/T377904 (10SHust) 03NEW [01:00:49] 06Fundraising-Backlog, 10FR-donorservices: Civi and acoustic odd merging behavior - https://phabricator.wikimedia.org/T377904#10252554 (10SHust) [01:01:07] (03PS1) 10Eileen: Attempt to follow up on loading change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082316 [01:21:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: target_contact_id is not valid : 61146956 - https://phabricator.wikimedia.org/T377903#10252574 (10Eileenmcnaughton) it's stalled on that contact - probably it got deleted & there is no merge record to tell it where to go - I guess we could maybe track the... [01:24:36] wfan: if you are ok to +2 these we can see if the upstream code helps... https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1082316 [01:24:36] The top one passes so they will all merge if all +2d [01:24:36] - the 3 in the middle are re-applying our patches & the top one is to make the tests pass (I'm feeding that back upstream) [01:26:26] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: silverpop_emails_check_data_file_upload - Out of date data_upload request found - https://phabricator.wikimedia.org/T377845#10252578 (10nisrael) @Eileenmcnaughton just to confirm, does this mean that we are we safe to continue sending Acoustic emails? [01:33:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: silverpop_emails_check_data_file_upload - Out of date data_upload request found - https://phabricator.wikimedia.org/T377845#10252580 (10Eileenmcnaughton) Yes - this is the result of the latest upload - which looks fine https://cloud.goacoustic.com/campaig... [01:33:43] or ejegg if youa re back... [01:40:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: silverpop_emails_check_data_file_upload - Out of date data_upload request found - https://phabricator.wikimedia.org/T377845#10252582 (10nisrael) Thank you @Eileenmcnaughton [01:40:59] (03PS1) 10Ejegg: Allow sending a note along with a cancellation [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082320 (https://phabricator.wikimedia.org/T373947) [01:56:06] we have had a speed up - 266 in the latest run [02:10:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Slow thank you email - https://phabricator.wikimedia.org/T377906 (10Eileenmcnaughton) 03NEW [02:13:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Slow thank you email - https://phabricator.wikimedia.org/T377906#10252613 (10Eileenmcnaughton) When we pushed out the 5.79 security update there were some changes made upstream that *might* help/hurt that we did not depl... [02:30:24] RECOVERY - check_missing_thank_yous on frdb1005 is OK: OK missing_thank_yous=1388 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [02:43:21] wahoo ^ [03:48:46] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10252637 (10Papaul) 05Open→03Resolved This is complete [03:48:53] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10252634 (10Papaul) 05Open→03Resolved This is complete. [04:16:37] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082322 [04:17:32] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082322 (owner: 10Eileen) [04:18:36] !log civicrm upgraded from de642bea to ce44ce45 [04:18:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:13:21] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10253170 (10cmooney) >>! In T377381#10252289, @Jclark-ctr wrote: > @cmooney Step 1: Firewall Installation... [12:22:01] (03CR) 10Damilare Adedoyin: [C:03+2] Attempt to follow up on loading change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082316 (owner: 10Eileen) [13:00:13] 06Fundraising-Backlog: Civicrm server /srv over 80% full, it's time to purge unnecessary data. - https://phabricator.wikimedia.org/T377946 (10Jgreen) 03NEW [13:00:33] 06Fundraising-Backlog: Civicrm server /srv over 80% full, it's time to purge unnecessary data. - https://phabricator.wikimedia.org/T377946#10253887 (10Jgreen) p:05Triage→03High [13:02:01] (03PS3) 10Damilare Adedoyin: Map Trustly/Gravy ACH "processing" status to Smashpig "Complete" status to enable immediate import to Civi for DR visibility. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1081180 (https://phabricator.wikimedia.org/T377376) [13:02:34] (03PS7) 10Damilare Adedoyin: Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) [13:04:16] fr-tech please can I get reviews on these patches so we can test them out before next monday tests [13:04:51] (03PS4) 10Damilare Adedoyin: Do capture only for Gravy Trustly recurring charge [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1079311 (https://phabricator.wikimedia.org/T371810) [13:05:04] I'm wrapping up some of my own patches dami but once done I'll check in on yours if no one else gets to 'em first [13:09:55] TY! [13:11:42] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Paypal - https://phabricator.wikimedia.org/T376469#10253937 (10jgleeson) [13:13:08] fr-tech I have two school runs today. Tilly has her Halloween disco after school so I'll be out between 3-5 today (1st 3:15, 2nd 4:15) and should be back for standup [13:27:19] Halloween disco! [13:27:25] looking damilare [13:27:26] :) [13:27:39] thanks ejegg [13:33:35] damilare: would that block - if ( isset( $params['recurring_payment_token'] ) ) { [13:33:49] make sense to go in the addRecurringParams function? [13:33:58] or is addRecurringParams only for first-time recurrings? [13:35:26] checking [13:36:30] or no addRecurringParams isn't jus for first-time recurrings, infact that block can go in there as weell [13:38:40] oh* just* [13:38:50] I'd push up a patch with the changes [13:40:26] (03PS5) 10Damilare Adedoyin: Do capture only for Gravy Trustly recurring charge [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1079311 (https://phabricator.wikimedia.org/T371810) [13:40:41] thanks ejegg [13:42:20] (03CR) 10Ejegg: [C:03+2] "Looks good, reduces complexity. Thanks!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1079311 (https://phabricator.wikimedia.org/T371810) (owner: 10Damilare Adedoyin) [13:42:49] (03Merged) 10jenkins-bot: Do capture only for Gravy Trustly recurring charge [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1079311 (https://phabricator.wikimedia.org/T371810) (owner: 10Damilare Adedoyin) [13:42:49] k, then there's the ACH one in SmashPig too, it looks like? [13:43:12] yes [13:43:33] that just allows us send ty email on ACH capture_pending status [13:43:48] "processing" [13:44:07] Ah OK. So is that different from what we do with iDEAL? [13:44:28] I think with iDEAL we wait for the IPN or the audit to actually record it in Civi, right? [13:45:26] Ah right, but Trustly gives us a realtime response about whether they actually succeeded in signing in to the bank [13:45:38] so it's pretty likely that it will complete [13:45:41] exactly [13:45:51] while iDEAL is just a redirect away and they might not get back to our site [13:45:58] OK, I guess that makes sense [13:46:00] this just gives DR more visibility [13:47:22] Let me check if any of the last week test payments have gotten it [13:51:15] hmm, Christine's transaction is marked settled on Trustly but failed on Gravy [13:51:28] oh weird [13:52:12] "Approval incomplete" [13:54:34] 10fundraising-tech-ops: Revise our mariadb grants deployment scheme. - https://phabricator.wikimedia.org/T377954 (10Jgreen) 03NEW [13:55:12] 10fundraising-tech-ops: Revise our mariadb grants deployment scheme. - https://phabricator.wikimedia.org/T377954#10254152 (10Jgreen) 05Open→03In progress p:05Triage→03Medium a:03Jgreen [13:56:33] (03PS2) 10Jgleeson: Add new Payment Orchestrator Reconciliation ID field. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082064 (https://phabricator.wikimedia.org/T374144) [13:57:20] 10fundraising-tech-ops: Revise our mariadb grants deployment scheme. - https://phabricator.wikimedia.org/T377954#10254165 (10Jgreen) [14:00:55] 10fundraising-tech-ops: Revise our mariadb grants deployment scheme. - https://phabricator.wikimedia.org/T377954#10254172 (10Jgreen) [14:03:25] (03PS1) 10Jgleeson: Fix DonationQueueTest dataProvider case for Backend processor fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082479 [14:13:10] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10254248 (10Jgreen) [15:21:00] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Slow thank you email - https://phabricator.wikimedia.org/T377906#10254604 (10greg) It caught back up around 7:30 UTC today (with a weird plateau?) and then started growing again when the 14:00 UTC campaign started: {F57... [15:23:12] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254615 (10cmooney) [15:24:01] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254623 (10cmooney) [15:41:51] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10254748 (10cmooney) [15:44:09] (03PS4) 10Damilare Adedoyin: Map Trustly/Gravy ACH "processing" status to Smashpig "Complete" status to enable immediate import to Civi for DR visibility. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1081180 (https://phabricator.wikimedia.org/T377376) [15:44:12] (03CR) 10Ejegg: [C:03+2] Map Trustly/Gravy ACH "processing" status to Smashpig "Complete" status to enable immediate import to Civi for DR visibility. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1081180 (https://phabricator.wikimedia.org/T377376) (owner: 10Damilare Adedoyin) [15:44:38] (03Merged) 10jenkins-bot: Map Trustly/Gravy ACH "processing" status to Smashpig "Complete" status to enable immediate import to Civi for DR visibility. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1081180 (https://phabricator.wikimedia.org/T377376) (owner: 10Damilare Adedoyin) [15:56:39] (03CR) 10Ejegg: [C:03+2] Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) (owner: 10Damilare Adedoyin) [16:04:57] (03Abandoned) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082305 (owner: 10Ejegg) [16:09:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Remove blank space in Gravy billing addresses - https://phabricator.wikimedia.org/T377979 (10jgleeson) 03NEW [16:34:20] 10fundraising-tech-ops, 10VPS-Projects, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10255182 (10Dwisehaupt) Hi, sorry about this. I did make progress and then promptly got placed on a jury where I have been for the last month. I'll plan to l... [16:53:07] ok looks like we are getting some ach transactions from last week in Civi now [16:54:08] here's ejegg transaction: https://civicrm.wikimedia.org/civicrm/contact/view/contribution?reset=1&id=113422692&cid=2241615&action=view&context=contribution&selectedChild=contribute [16:56:12] the queue delay would affect the TY mails going out, other than that everything seems ok [16:56:40] I'd push that patch out to get it imported without the wait [16:57:01] 10fundraising-tech-ops: Deactivate fundraising accounts for nnichols - https://phabricator.wikimedia.org/T377995 (10Dwisehaupt) 03NEW [16:57:48] (03Merged) 10jenkins-bot: Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) (owner: 10Damilare Adedoyin) [17:01:19] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10255334 (10cmooney) Just a note to say that fundraising no longer use any VM infra, so every assigned IP I believe belongs to just a single server. [17:01:27] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10255336 (10cmooney) [17:06:14] (03PS1) 10Damilare Adedoyin: Update smashpig version to 0.8.16 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082517 [17:07:16] 10fundraising-tech-ops, 10BDC-Implementation: [dbt] unblock packages install - https://phabricator.wikimedia.org/T377369#10255372 (10Irisbdc) Going with the proxy approach. [17:08:18] 10fundraising-tech-ops, 10BDC-Implementation: [Metabase] Investigate packaging, install, security monitoring. - https://phabricator.wikimedia.org/T375030#10255376 (10Irisbdc) A version is running. Putting in backburner for now. [17:09:55] (03CR) 10Damilare Adedoyin: [C:03+2] Update smashpig version to 0.8.16 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082517 (owner: 10Damilare Adedoyin) [17:15:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Remove blank space in Gravy billing addresses - https://phabricator.wikimedia.org/T377979#10255425 (10jgleeson) [17:32:55] (03PS1) 10Jgleeson: Remove blank space in Gravy billing addresses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082523 (https://phabricator.wikimedia.org/T377979) [17:33:00] 10fundraising-tech-ops, 10BDC-Implementation: [dbt] unblock packages install - https://phabricator.wikimedia.org/T377369#10255472 (10Jgreen) 05Open→03Resolved Great! Closing the task. [17:38:51] (03Merged) 10jenkins-bot: Update smashpig version to 0.8.16 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1082517 (owner: 10Damilare Adedoyin) [17:50:05] (03CR) 10Jgleeson: "I need to smoke test this with a sandbox transaction" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082523 (https://phabricator.wikimedia.org/T377979) (owner: 10Jgleeson) [17:52:53] 03Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog: Add address fields for all Gravy 3D Secure countries - https://phabricator.wikimedia.org/T377846#10255549 (10jgleeson) Moving this to blocked/on-hold as Gravy suggest we might not need it once {T377979} is deployed [17:54:39] damilare: does this look good to you WRT to removing that space? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1082523 [17:54:51] is there anywhere else I've missed [17:56:38] checking [18:11:31] jgleeson: yeah that is all the areas we need to update. However, looks like some of these fields are actually required: https://docs.gr4vy.com/reference/transactions/new-transaction#:~:text=number%20standard.-,buyer.billing_details.address,-object [18:12:06] like city/country/postal_code/state/line1 [18:13:54] hmm so I just smoke tested a few payments for AU, IE and US with those changes and I didn't run into any errors [18:14:14] the last 3 here https://wikimedia.gr4vy.app/merchants/default/transactions [18:14:19] 06Fundraising-Backlog, 10FR-donorservices, 07payments-orchestration: Gr4vy console setting: enable viewing of name in Buyers tab - https://phabricator.wikimedia.org/T378007 (10MBeat33) 03NEW [18:14:26] arg wrong URL [18:14:39] https://sandbox.wikimedia.gr4vy.app/merchants/default/transactions <-- here evem [18:15:28] ahh cool then [18:15:37] I do see the required tag tho [18:15:47] also the mapToCreateDonorRequest function is just like our sprint name [18:15:48] maybe we should flag that with philip [18:15:52] yep [18:16:00] lol [18:17:01] yep I can +2 that, or do you still want to make changes? [18:17:17] no if you says it's good it's good! [18:17:24] ty [18:17:59] (03CR) 10Damilare Adedoyin: [C:03+2] Remove blank space in Gravy billing addresses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082523 (https://phabricator.wikimedia.org/T377979) (owner: 10Jgleeson) [18:18:12] np LGTM [18:18:28] (03Merged) 10jenkins-bot: Remove blank space in Gravy billing addresses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082523 (https://phabricator.wikimedia.org/T377979) (owner: 10Jgleeson) [18:26:01] (03PS3) 10Jgleeson: Capture Payment Orchestrator Reconciliation ID. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1082062 (https://phabricator.wikimedia.org/T374144) [18:28:09] damilare|brb: I just updated GravyGetLatestPaymentStatusResponseFactory in that smashpig patch as requested. When updating the corresponding test, I noticed we're using a create-transaction fixture here https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/44d7261ccd46a0e092e35936fb6a1bc4f3aaedb8/PaymentProviders/Gravy/Tests/phpunit/PaymentProviderTest.php#L75 [18:29:26] I imagine they are similar API responses but we might wanna confirm and add a new file for the status check one [18:32:35] oh right, I'm finising early tonight. Bayern Munich are playing Barcelona in the Champions League and it's being billed as clash of the Titans so I'm spreading today's last hour over thursday & friday so I can watch it :) [18:54:32] (03CR) 10Ejegg: [V:03+2 C:03+2] Latest upstream 5.79rc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082311 (owner: 10Eileen) [18:55:08] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082311 (owner: 10Eileen) [18:55:27] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082313 (https://phabricator.wikimedia.org/T374063) (owner: 10Eileen) [18:56:04] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082314 (https://phabricator.wikimedia.org/T375039) (owner: 10Eileen) [18:56:33] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082315 (https://phabricator.wikimedia.org/T353971) (owner: 10Eileen) [19:10:22] PROBLEM - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2245 [critical =2000] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [19:12:23] ACKNOWLEDGEMENT - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=2245 [critical =2000] Dwisehaupt known and being investigated. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [19:20:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Slow thank you email - https://phabricator.wikimedia.org/T377906#10255957 (10Eileenmcnaughton) Command to render a bunch through the UI - to run render a bunch of times in a process echo '{"orderBy":{"id":"DESC"},"lim... [19:27:04] (03Merged) 10jenkins-bot: Attempt to follow up on loading change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082316 (owner: 10Eileen) [19:27:51] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082544 [19:27:55] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082544 (owner: 10Ejegg) [19:28:47] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082544 (owner: 10Ejegg) [19:35:28] !log civicrm upgraded from ce44ce45 to e787e5f2 [19:35:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:46:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06MediaWiki-Platform-Team, 10MediaWiki-Vendor, and 5 others: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#10256039 (10hashar) 05Resolved→03Open This is not resolved since https://ge... [19:51:08] (03PS1) 10Eileen: wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082550 [20:36:39] (03PS1) 10Eileen: Revert cache clear [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082556 [20:37:56] (03CR) 10Ejegg: [C:03+2] Revert cache clear [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082556 (owner: 10Eileen) [20:38:00] (03CR) 10Ejegg: [V:03+2 C:03+2] Revert cache clear [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082556 (owner: 10Eileen) [20:39:23] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082558 [20:39:34] (03CR) 10Eileen: [V:03+2 C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1082558 (owner: 10Eileen) [20:40:54] !log civicrm upgraded from e787e5f2 to 1c6c4e08 [20:40:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:48:48] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256380 (10cmooney) [20:58:22] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256417 (10cmooney) [21:02:29] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256437 (10Jclark-ctr) @cmooney all cables have been connected for Step 2: Initial cabling for the new de... [21:05:17] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10256451 (10cmooney) >>! In T377381#10256437, @Jclark-ctr wrote: > @cmooney all cables have been connected... [21:33:39] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2425Q2 maintenance window (Nov 4-8th, 2024) - https://phabricator.wikimedia.org/T337583#10256541 (10Dwisehaupt) [22:30:24] RECOVERY - check_missing_thank_yous on frdb1005 is OK: OK missing_thank_yous=1289 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous