[00:01:11] FIRING: NodeDown: Node civi1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:02:39] RESOLVED: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [00:11:09] RESOLVED: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [00:12:43] RESOLVED: RedisQueueSize: Redis Queue contribution_tracking is high: [4326] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [00:18:41] RESOLVED: NodeDown: Node civi1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:21:13] RESOLVED: RedisQueueSize: Redis Queue contribution_tracking is Critical: [4326] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [01:03:08] (03PS1) 10Ejegg: Add invoice ID to refund contribution form [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1291025 (https://phabricator.wikimedia.org/T421277) [01:35:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen transaction - incorrect backend_gateway_txn_id - https://phabricator.wikimedia.org/T427028 (10Eileenmcnaughton) 03NEW [01:36:12] (03PS1) 10Eileen: Extend repair date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1291058 (https://phabricator.wikimedia.org/T427028) [01:37:49] 06Fundraising-Backlog, 10Phabricator-Bot-Requests: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459#11947159 (10AKanji-WMF) a:03Aklapper Following up on this, I see {T425118} was assigned to @Aklapper - I'll do the same here, but please let me know if there'... [01:43:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Paypal transaction - incorrect backend_gateway_txn_id - https://phabricator.wikimedia.org/T427029 (10Eileenmcnaughton) 03NEW [01:55:36] (03CR) 10Lars SG: [C:03+2] Extend repair date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1291058 (https://phabricator.wikimedia.org/T427028) (owner: 10Eileen) [01:56:00] (03CR) 10Lars SG: [C:03+2] Add invoice ID to refund contribution form [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1291025 (https://phabricator.wikimedia.org/T421277) (owner: 10Ejegg) [01:59:41] thanks larssandergreen! [02:01:04] I hope you're home [02:01:43] Hanging out in the airport, but Calgary, so I'm close. [02:03:02] awesome to hear larssandergreen [02:09:40] woo larssandergreen [02:34:30] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen transaction - incorrect backend_gateway_txn_id - https://phabricator.wikimedia.org/T427028#11947175 (10Ejegg) Sadly we didn't record the raw API response in our job logs. 2026-05-13 00:27:44,213 Charging contribution_recur id: 229... [02:50:36] have a good weekend, all! [05:25:09] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1291345 (owner: 10L10n-bot) [07:53:57] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1291747 (owner: 10L10n-bot) [08:12:37] (03Abandoned) 10Raimond Spekking: Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290594 (owner: 10L10n-bot) [08:17:25] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1291747 (owner: 10L10n-bot) [08:35:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:20:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:50:44] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07fr-current-sprint: Donor Portal documentation - https://phabricator.wikimedia.org/T410588#11947598 (10Damilare) The documentation has been added to WikiTech and a link was added to collab. [10:07:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: no email_checksum if update email too - https://phabricator.wikimedia.org/T426113#11947627 (10AnnWF) We only need email_checksum when we update email and need to send verification email, for this case, changed email from A to B, then before B get verified... [10:30:59] (03PS1) 10Damilare Adedoyin: Rename mwGetRequest to mwRequest [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1291942 [10:35:09] (03CR) 10CI reject: [V:04-1] Rename mwGetRequest to mwRequest [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1291942 (owner: 10Damilare Adedoyin) [10:37:06] (03CR) 10Damilare Adedoyin: "Recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1291942 (owner: 10Damilare Adedoyin) [10:38:31] (03PS1) 10Wfan: move setup script echo to readme for all application urls [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1291945 [11:25:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:32:43] (03PS1) 10Wfan: add name split for recurring payment if any empty [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1291963 (https://phabricator.wikimedia.org/T424766) [11:33:12] (03CR) 10CI reject: [V:04-1] add name split for recurring payment if any empty [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1291963 (https://phabricator.wikimedia.org/T424766) (owner: 10Wfan) [11:35:39] (03PS2) 10Wfan: add name split for recurring payment if any empty [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1291963 (https://phabricator.wikimedia.org/T424766) [11:45:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:25:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:26:18] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Recurring Token Migrate - Adyen CC - https://phabricator.wikimedia.org/T418759#11948095 (10Damilare) I sent the first set of recurrings for a test export to Adyen today. The list can be found on this... [13:10:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [16.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:07:10] (03PS4) 10Damilare Adedoyin: DonorPortal: Add text for legacy paypal to explain why they currently cannnot be managed on the DonorPortal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) [14:09:08] (03CR) 10Damilare Adedoyin: "Thanks for the CR, Done!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) (owner: 10Damilare Adedoyin) [14:22:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:32:51] 06Fundraising-Backlog, 10Phabricator-Bot-Requests: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459#11948520 (10Aklapper) Hi, sorry, this task fell between the cracks somehow. :( Please see https://www.mediawiki.org/wiki/Phabricator/Bots#Acquiring_a_bot for req... [14:40:57] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824#11948571 (10Jhancock.wm) 05Open→03Resolved [15:12:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [15:41:29] (03CR) 10Peterxy12: [C:03+1] "recheck" [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1291345 (owner: 10L10n-bot) [15:49:45] 10Fundraising analytics stack: install jupyterhub to facilitate sharing notebooks - https://phabricator.wikimedia.org/T427079 (10AStein-WMF) 03NEW [16:03:23] 10Fundraising analytics stack: install jupyterhub to facilitate sharing notebooks - https://phabricator.wikimedia.org/T427079#11948959 (10AStein-WMF) [16:34:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: missing partial name from 3rd party response - https://phabricator.wikimedia.org/T424766#11949098 (10AnnWF) Steps to reproduce: Donate with apple pay with full name update full name with blank firstname recurring charge failed due to... [17:04:50] (03PS1) 10Wfan: use warning than exception if no checksum but email diff [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1292061 (https://phabricator.wikimedia.org/T426113) [17:49:56] (03CR) 10Cstone: [C:03+2] use warning than exception if no checksum but email diff [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1292061 (https://phabricator.wikimedia.org/T426113) (owner: 10Wfan) [17:58:13] (03CR) 10Wfan: [V:03+2 C:03+2] move setup script echo to readme for all application urls [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1291945 (owner: 10Wfan) [18:12:20] (03Merged) 10jenkins-bot: use warning than exception if no checksum but email diff [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1292061 (https://phabricator.wikimedia.org/T426113) (owner: 10Wfan) [18:43:11] FIRING: NodeDown: Node fransw2001 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:43:47] FIRING: HAProxyExporterDown: HAProxy on fransw2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [18:48:11] RESOLVED: NodeDown: Node fransw2001 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:48:47] RESOLVED: HAProxyExporterDown: HAProxy on fransw2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [19:39:11] FIRING: NodeDown: Node frbast1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:44:11] RESOLVED: NodeDown: Node frbast1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown