[00:00:41] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287500 (owner: 10Eileen) [00:02:05] !log civicrm upgraded from 6d8ce7a3 to 6a2258ff [00:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [00:03:24] (03CR) 10CI reject: [V:04-1] Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [00:12:54] (03PS5) 10Wfan: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [00:13:03] (03PS4) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [00:34:11] (03CR) 10CI reject: [V:04-1] Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [00:34:16] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [00:39:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [00:57:31] (03PS1) 10Eileen: More detail when the zipping fails [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287505 [00:59:35] (03CR) 10Eileen: [C:03+2] "self+2 as this is debug - which we agreed is OK to self +2" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287505 (owner: 10Eileen) [01:04:38] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11924049 (10Dwisehaupt) [01:21:21] (03Merged) 10jenkins-bot: More detail when the zipping fails [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287505 (owner: 10Eileen) [01:28:37] eileen: so far all the adyen recurring charges for today's run seem to be coming in with backend txn id == gateway txn id [01:29:06] and == auth id [01:32:24] yay [02:00:07] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287511 [02:00:23] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287511 (owner: 10Eileen) [02:01:28] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287511 (owner: 10Eileen) [03:58:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:13:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:31:31] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1287566 (owner: 10L10n-bot) [06:00:57] FIRING: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [06:15:57] RESOLVED: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [09:03:40] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1286004 (owner: 10L10n-bot) [09:03:41] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1286588 (owner: 10L10n-bot) [09:04:05] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_45) - 10https://gerrit.wikimedia.org/r/1287131 (owner: 10L10n-bot) [09:05:13] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1287566 (owner: 10L10n-bot) [09:12:08] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change last donation amount, last donation original currency and last donation currency from foundation to all funds - https://phabricator.wikimedia.org/T426189#11924591 (10Lars) Now this needs to be backfilled [09:14:10] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#11924621 (10Lars) The above has been done, including deleting fields. Keeping this open for the remaining items. [09:14:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Delete disabled fields from wmf_contribution_extra - https://phabricator.wikimedia.org/T420301#11924629 (10Lars) [09:15:14] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Delete disabled fields from wmf_contribution_extra - https://phabricator.wikimedia.org/T420301#11924631 (10Lars) a:03Lars [09:26:19] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Join contribution tracking to contributions in SK so we can filter without adding a join manually - https://phabricator.wikimedia.org/T412453#11924673 (10Lars) 05Open→03Invalid This had already been done when I create... [09:40:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [09:40:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [09:40:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [09:45:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [09:45:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [09:45:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [09:58:48] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Join contribution tracking to contributions in SK so we can filter without adding a join manually - https://phabricator.wikimedia.org/T412453#11924780 (10Lars) 05Invalid→03Op... [09:59:02] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Join contribution tracking to contributions in SK so we can filter without adding a join manually - https://phabricator.wikimedia.org/T412453#11924785 (10Lars) a:03Lars [10:00:10] (03PS1) 10Lars SG: Add implicit join for contribution trackings from contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287830 (https://phabricator.wikimedia.org/T412453) [10:13:55] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11924801 (10Lars) [10:17:20] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11924823 (10Lars) @MBeat33 I've added the risk score to the searchkit, as it was in the report. Does that look right to you? Turns out to have bee... [10:17:51] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11924827 (10Lars) a:03Lars [11:24:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:54:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:16:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:31:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [13:17:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Restore description_of_stock field values incorrectly deleted. - https://phabricator.wikimedia.org/T426417 (10Lars) 03NEW [13:21:00] (03PS1) 10Lars SG: Restore description_of_stock field values incorrectly deleted [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287871 (https://phabricator.wikimedia.org/T426417) [13:24:20] fr-tech so Krinkle did give some feedback on that CentralNotice hook - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/1282396/1#message-378debc6494c231be3773f9ddf0b5b00e96c2686 [13:25:41] (03CR) 10Jgleeson: [C:03+2] "This looks good to me!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287871 (https://phabricator.wikimedia.org/T426417) (owner: 10Lars SG) [13:25:55] making the point that a) there's no dependency between the two scripts, so there's no way to ensure the test kitchen script can subscribe before the hook fires [13:27:58] and b) that hook.fire() is async so state.failCampaign() might not be called before the next iteration [13:28:38] I haven't though of any simple fix just yet [13:31:04] (03PS5) 10Ejegg: Remove special subscr_id for gravy/paypal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1275995 (https://phabricator.wikimedia.org/T423638) [13:40:18] ejegg: I expect with my core fix landed, and them using cancelling from SiteNoticeBefore hook, the CN fix for that should suffice to close the task. Optimistic, we'll hear soon I hope. [13:41:24] (03PS2) 10Lars SG: Restore description_of_stock field values incorrectly deleted [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287871 (https://phabricator.wikimedia.org/T426417) [13:41:29] (03PS2) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [13:43:10] yep Krinkle, that should fix their issue [13:43:32] I was hoping we could use that hook for another project though [13:43:47] There is talk of using TestKitchen to optimize banner selection [13:43:49] (03CR) 10CI reject: [V:04-1] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [13:44:29] instrumenting clickthroughs to payments-wiki so they're saved in the TestKitchen data [13:45:12] then doing something at that selection point for TestKitchen to replace the banner array in the campaign (or fail the campaign) [13:46:20] so I'll have to think of a new strategy for that [13:46:41] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11925549 (10krobinson) We... [13:49:44] (03PS3) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [13:50:13] (03Abandoned) 10Ejegg: Update smashpig to v0.9.6.4 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1147793 (owner: 10Damilare Adedoyin) [13:51:37] (03CR) 10CI reject: [V:04-1] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [13:54:17] (03PS6) 10Lars SG: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) [13:54:17] (03PS5) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [13:55:01] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [13:57:20] (03PS6) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [13:57:49] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [13:58:33] (03PS3) 10Lars SG: Setting to exclude location types from Email::Clean [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282926 (https://phabricator.wikimedia.org/T425195) [13:58:43] (03PS4) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [13:59:01] (03PS7) 10Lars SG: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) [13:59:10] oh we should add the php-ast extension to our dev images so we can run phan locally [14:00:52] huh, looking at the amazon code I'm wondering if we really want to save it [14:01:33] we're not doing the same normalization at the SmashPig level on requests or responses that we do with all the rest of the providers [14:02:24] The DonationInterface code is just calling the SDK methods directly [14:03:37] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11925591 (10MBeat33) Thank you so much, @Lars I just tested the SK, and I have two followup ques... [14:04:07] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11925592 (10AStein-WMF) done! [14:04:32] (03CR) 10CI reject: [V:04-1] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [14:04:47] (03CR) 10Lars SG: "I'll rebase this after the changes before it have been merged, since we don't want to run it before those are through." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [14:17:16] (03PS5) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [14:20:21] (03CR) 10CI reject: [V:04-1] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [14:25:07] (03PS6) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [14:31:11] FIRING: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:32:06] (03CR) 10Ejegg: [V:03+2] build: Updating postcss to 8.5.14 [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1280168 (owner: 10Libraryupgrader) [14:32:09] (03CR) 10Ejegg: [V:03+2] build: Updating postcss to 8.5.14 [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1280167 (owner: 10Libraryupgrader) [14:36:11] FIRING: NodeDown: Node fran2001 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:37:36] FIRING: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [14:38:36] (03PS7) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [14:38:39] (03CR) 10Ejegg: [C:03+2] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [14:39:16] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11925722 (10Lars) @MBeat33 I think I understand now, does that look right to you? [14:40:26] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1287894 [14:40:31] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1287894 (owner: 10Ejegg) [14:40:50] (03Merged) 10jenkins-bot: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [14:42:52] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1287894 (owner: 10Ejegg) [14:46:11] RESOLVED: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:46:11] RESOLVED: NodeDown: Node fran2001 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:46:56] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11925752 (10MBeat33) Thank you @Lars The one line per transaction in the export is great. Just to... [14:47:36] RESOLVED: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [15:00:11] FIRING: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:00:52] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11925792 (10Lars) @MBeat33 Yes, that's the same field as in the Fredge report. If you wanted to, y... [15:05:11] RESOLVED: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:11:53] (03PS1) 10Ejegg: Only instantiate cache when needed to send failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287903 [15:13:07] fr-tech our local dev setup actually differs slightly from prod in the caching config [15:13:20] we use the prediscache for SmashPig in all containers [15:13:42] while production uses the MW cache wrapper for caching under payments [15:14:05] anyway, when I tried using the MW cache wrapper locally I found it was throwing warnings [15:14:18] which might be errors under the next LTS version of mediawiki [15:15:02] because it's initializing the cache object before the MW service locator is ready [15:15:21] anyway, this patch seems to fix it: https://gerrit.wikimedia.org/r/1287903 [15:16:31] besides that, the library upgrade seems totally fine locally [15:17:03] so I'd like to do an early-friday payments deploy to get through the dependency update chain [15:18:35] (03PS1) 10Ejegg: Update composer files [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1287904 [15:18:38] (03CR) 10Ejegg: [C:03+2] Update composer files [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1287904 (owner: 10Ejegg) [15:29:29] (03Merged) 10jenkins-bot: Update composer files [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1287904 (owner: 10Ejegg) [15:35:05] !log payments-wiki upgraded from cf9ec80b to 1a056dc0 [15:35:06] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [15:35:11] FIRING: NodeDown: Node frdb2003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:37:16] (03CR) 10Jgleeson: [C:03+2] Only instantiate cache when needed to send failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287903 (owner: 10Ejegg) [15:37:23] sounds good ejegg [15:37:35] so far so good [15:37:38] thanks for the CR! [15:37:49] (03Merged) 10jenkins-bot: Only instantiate cache when needed to send failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287903 (owner: 10Ejegg) [15:40:11] FIRING: NodeDown: Node frdb2003 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:40:57] FIRING: MariadbNoReplicas: No MariaDB replicas connected to : frdb1007.frack.eqiad.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [15:53:27] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11926075 (10MBeat33) Thank you for confirming, and the filter option. When a great tool gets even... [15:54:44] (03CR) 10Ejegg: "Looks solid and understandable. One typo in a comment." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1273814 (https://phabricator.wikimedia.org/T421662) (owner: 10Krinkle) [15:58:30] oh I should update donor portal to the latest too [15:59:05] !log donorwiki upgraded from 26f5451a to 1a056dc0 [15:59:06] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [16:25:11] RESOLVED: NodeDown: Node frdb2003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:25:11] RESOLVED: NodeDown: Node frdb2003 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:25:57] RESOLVED: MariadbNoReplicas: No MariaDB replicas connected to : frdb1007.frack.eqiad.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [16:31:41] FIRING: [2x] NodeDown: Node fran2001 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:35:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: foreign currency block the recurring paypal - https://phabricator.wikimedia.org/T426447 (10AnnWF) 03NEW [16:37:11] FIRING: NodeDown: Node fran2001 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:38:10] FIRING: [4x] PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:39:35] FIRING: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [16:42:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Recurring paypal failed and currency conversion show wrong in civi - https://phabricator.wikimedia.org/T426447#11926395 (10AnnWF) [16:48:29] 10Fundraising analytics stack: create DNS aliases for user db connections - https://phabricator.wikimedia.org/T426448 (10AStein-WMF) 03NEW [17:03:44] (03PS4) 10Ejegg: DonorPortal: Update mocked tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1270044 (owner: 10Damilare Adedoyin) [17:03:55] (03CR) 10Ejegg: [C:03+2] DonorPortal: Update mocked tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1270044 (owner: 10Damilare Adedoyin) [17:04:36] FIRING: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [17:07:30] (03Merged) 10jenkins-bot: DonorPortal: Update mocked tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1270044 (owner: 10Damilare Adedoyin) [17:53:04] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Batch refund action for unintended recurring contribution - https://phabricator.wikimedia.org/T426350#11926710 (10Ejegg) Instead of a separate action, maybe this could be another change to the cancel modal? e.g. when you select 'u... [17:56:30] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10FR-Smashpig: Create and populate properties for auth and capture ID on SmashPig responses - https://phabricator.wikimedia.org/T421380#11926718 (10Ejegg) This is done for Gravy and Adyen... [17:56:41] RESOLVED: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:57:11] RESOLVED: NodeDown: Node fran2001 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:59:36] RESOLVED: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [17:59:36] RESOLVED: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [18:25:53] 06Fundraising-Backlog, 10Phabricator-Bot-Requests: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459 (10AKanji-WMF) 03NEW [18:43:53] (03PS1) 10Ejegg: Delete several more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287932 (https://phabricator.wikimedia.org/T111677) [18:45:11] FIRING: [2x] NodeDown: Node frqueue1005 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:50:11] RESOLVED: [2x] NodeDown: Node frqueue1005 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:56:41] FIRING: [3x] NodeDown: Node frqueue1005 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:01:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [19:01:41] RESOLVED: [3x] NodeDown: Node frqueue1005 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:03:44] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, 0725 Dec Q3 Planning: Civi: enable search by merchant reference # for failed donations - https://phabricator.wikimedia.org/T402425#11926884 (10AMJohnson) Just throwing in a newer example of an error c... [19:16:53] (03PS1) 10Ejegg: Delete more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287936 (https://phabricator.wikimedia.org/T111677) [19:18:27] ok, that's about 100 more message deletions [19:19:03] !log redis swap complete from frqueue1003 to frqueue1005 [19:19:04] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [19:36:48] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11927042 (10CHudson-WMF) Hi, @Ejegg--I've started working on this and I noticed that there are some activities for recurring retries. Do you know under what circumstances the... [19:40:49] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11927058 (10Ejegg) @CHudson-WMF that sounds like the Adyen auto-rescue flow, where they retry a charge for us over a couple weeks. We only activate that for recurring installm... [19:42:17] (03CR) 10Cstone: [C:03+2] Delete several more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287932 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [19:44:02] (03Merged) 10jenkins-bot: Delete several more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287932 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [19:59:36] oh man, tamagotchi in a vape dies if you don't puff: https://futurism.com/neoscope/vape-tamagotchi-interview [20:02:42] oh huh, we don't actually use all those currency code translations as far as I can tell [20:22:45] (03PS1) 10Ejegg: Delete more unused messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287939 (https://phabricator.wikimedia.org/T111677) [20:25:28] ok, that's all the ones that are really obviously unused, except the currencies [20:25:58] I kind of want to add those currency names to the dropdown just cause we have htem [20:26:41] but it would bloat the strings for any client-side rendered version [20:33:10] FIRING: [4x] PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:58:10] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:58:48] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Batch refund action for unintended recurring contribution - https://phabricator.wikimedia.org/T426350#11927212 (10MBeat33) Thanks @Ejegg this touches on a few wider workflows with recurring cancellations, so I'm more subscribers f... [21:03:10] RESOLVED: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:32:47] (03CR) 10Cstone: [C:03+2] Delete more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287936 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [21:33:09] (03CR) 10Cstone: [C:03+2] Delete more unused messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287939 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [21:45:20] (03Merged) 10jenkins-bot: Delete more unused i18n messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287936 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [21:45:22] (03Merged) 10jenkins-bot: Delete more unused messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287939 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [22:57:11] FIRING: NodeDown: Node frbast2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:02:11] RESOLVED: NodeDown: Node frbast2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:33:11] FIRING: [3x] NodeDown: Node fran1002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:34:47] FIRING: HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [23:35:12] ^^^ these are expected with host reboots. they are coming back online. [23:37:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [23:38:11] RESOLVED: [3x] NodeDown: Node fran1002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:39:47] RESOLVED: HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [23:42:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning