[00:03:20] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: Decommission frack hosts: civi2001 - https://phabricator.wikimedia.org/T397380#10952576 (10Dwisehaupt) [00:04:22] 10fundraising-tech-ops, 06SRE: Q1:rack/setup/install frand200[12] - https://phabricator.wikimedia.org/T367804#10952580 (10Dwisehaupt) a:05Dwisehaupt→03None [00:05:55] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2025: Update sync-to-staging to also sync the angular files - https://phabricator.wikimedia.org/T396693#10952581 (10Dwisehaupt) Files are pulled into the offhost backup. Still need to update the refresh docu... [00:07:15] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2025: Update sync-to-staging to also sync the angular files - https://phabricator.wikimedia.org/T396693#10952582 (10Dwisehaupt) [00:09:21] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q1:rack/setup/install frnetmon1002, pay-lb1001, pay-lb1002 - https://phabricator.wikimedia.org/T369565#10952585 (10Dwisehaupt) 05Open→03Resolved @Jgreen Was able to bring the bond interface up on pay-lb1002 and it has survi... [00:13:27] 14Fundraising Sprint Interstitial ads halfway down the coaster hill, 14fundraising sprint Just in time roller coaster building, 14Fundraising Sprint Ketchup Flume Ride, 14fundraising sprint Legø Trampoline, and 3 others: Enable SSL for CiviCRM DB connectio... - https://phabricator.wikimedia.org/T280080#10952594 [03:30:30] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_42) - 10https://gerrit.wikimedia.org/r/1163950 (owner: 10L10n-bot) [04:43:38] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10952802 (10RKumar_WMF) Added to Gr4vy tracker. [07:11:29] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10952973 (10Volans) When editing netbox DNS records please always make sure to run the `sre.dns.netbox` cookbook as o... [07:12:57] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10952974 (10krobinson) Can I confirm if 3DS is standard in Canada? I don't feel like we normally see 3DS rejections from there. Is that part of the Adyen flow? [07:13:30] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10952976 (10Volans) There are currently changes to remove: ` frpig2001.mgmt.frack.codfw.wmnet pay-lvs2001.mgmt.frack.... [07:44:31] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10953070 (10krobinson) FYI this seems to also be affecting Australia. Yamini has come across 3 3DS AU declines in the past couple of hours in the queue. We just did sends to 6C earlier this... [08:30:39] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10953167 (10RKumar_WMF) As we have moved cards traffic to Adyen direct since yesterday (26th June) I am not sure if this is due to Gr4vy. Also, this has been raised to Gr4vy on slack channel.... [08:37:24] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10953179 (10Stevemunene) >>! In T397868#10952976, @Volans wrote: > There are currently changes to remove: > ` > frpig2001.... [12:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 51.94, 62.43, 34.54 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [12:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 51.40, 37.23 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [12:53:11] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.73, 3.53, 8.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [12:55:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 3.51, 3.78, 8.69 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:46:43] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10954132 (10krobinson) These seem to be from before the switch @RKumar_WMF - all are from 25th June or early on 26th. We have more examples from Australia too, so I don't think this is isola... [14:03:02] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10954210 (10Andrew) For whoever applies these pending changes... there are now also (non-urgent) pending changes for cloud... [14:45:32] (03CR) 10Jgleeson: [C:04-1] "What ID should we be using with this one?" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163912 (owner: 10Ejegg) [14:46:00] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163508 (https://phabricator.wikimedia.org/T397342) (owner: 10Wfan) [14:46:54] (03CR) 10Jgleeson: [C:03+2] "Exciting!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163458 (https://phabricator.wikimedia.org/T199346) (owner: 10Ejegg) [14:47:21] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163427 (https://phabricator.wikimedia.org/T397657) (owner: 10Wfan) [14:52:01] (03CR) 10Jgleeson: [C:03+2] "the box is bigger :)" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163071 (https://phabricator.wikimedia.org/T397251) (owner: 10Wfan) [15:05:17] (03Merged) 10jenkins-bot: Update AdvancedCriteria.tpl to default open search setting open [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163508 (https://phabricator.wikimedia.org/T397342) (owner: 10Wfan) [15:05:20] (03Merged) 10jenkins-bot: Expose psr/container interface for config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163458 (https://phabricator.wikimedia.org/T199346) (owner: 10Ejegg) [15:07:17] (03Merged) 10jenkins-bot: if checksum expired, do not send queue with hash validator [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163427 (https://phabricator.wikimedia.org/T397657) (owner: 10Wfan) [15:10:43] (03Merged) 10jenkins-bot: make the snoozed date selector year wider and viable for input [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163071 (https://phabricator.wikimedia.org/T397251) (owner: 10Wfan) [15:29:18] Thanks Jack, that snooze date could be a temporary update since they (upstream civi ppl) have a discussion of how to make the input filed visible like we have border to show input or they might want to add edit icon after the field. While anyway the one just merged would be good for us to use for while. [15:31:12] (03CR) 10Jgleeson: [C:03+2] "This looks like a great addition. I was wondering how we were persisting with the count at first, but then I realised you were reading in " [wikimedia/fundraising/process-control] - 10https://gerrit.wikimedia.org/r/1162944 (https://phabricator.wikimedia.org/T397630) (owner: 10Ejegg) [15:31:45] (03Merged) 10jenkins-bot: Allow suppressing failmail till N failures [wikimedia/fundraising/process-control] - 10https://gerrit.wikimedia.org/r/1162944 (https://phabricator.wikimedia.org/T397630) (owner: 10Ejegg) [15:38:27] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10954589 (10Dwisehaupt) Sorry about that. I ran the authdns-update but totally forgot about the cookbook. I've updated my... [15:43:21] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#10954596 (10Stevemunene) Thanks @Dwisehaupt [16:13:48] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission frack hosts: frban1001 - https://phabricator.wikimedia.org/T397869#10954736 (10VRiley-WMF) 05Open→03Resolved [16:14:08] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission frack hosts: frban1001 - https://phabricator.wikimedia.org/T397869#10954742 (10VRiley-WMF) Unracked and decommed this server [18:44:44] (03PS4) 10Damilare Adedoyin: WIP [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163874 [18:47:23] (03CR) 10CI reject: [V:04-1] WIP [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163874 (owner: 10Damilare Adedoyin) [19:35:46] (03PS1) 10Wfan: instead of update primary email via EPC add a email confirm step [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) [19:47:46] 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10955241 (10MBeat33) Piero stepped up: "I've reviewed the reported transactions from both Canada and Australia, and here are my findings: On June 25th, you saw a larger volume of donations f... [19:52:15] (03CR) 10CI reject: [V:04-1] instead of update primary email via EPC add a email confirm step [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) (owner: 10Wfan) [19:55:45] (03CR) 10Wfan: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) (owner: 10Wfan) [20:32:27] 10fundraising-tech-ops, 10decommission-hardware: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079 (10Dwisehaupt) 03NEW