[00:02:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [00:04:50] ^^^ re-silenced. it cleared when the puppet server was restarted. [00:12:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on civi2002.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [00:47:10] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on civi2002.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [01:03:11] FIRING: NodeDown: Node frbast1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [01:08:11] RESOLVED: NodeDown: Node frbast1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frbast1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [01:12:45] (03PS1) 10Eileen: Extend repair date ... again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287045 [05:41:30] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_45) - 10https://gerrit.wikimedia.org/r/1287131 (owner: 10L10n-bot) [06:01:57] FIRING: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [06:16:57] RESOLVED: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [08:53:04] (03CR) 10Lars SG: [C:03+2] Extend repair date ... again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287045 (owner: 10Eileen) [09:18:41] (03Merged) 10jenkins-bot: Extend repair date ... again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287045 (owner: 10Eileen) [09:26:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:46:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [10:55:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:05:41] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#11921208 (10Lars) We'll need to do the following as well, as we will lose the lifetime_usd_total field in this index: ` ALTER... [11:08:15] (03PS1) 10Lars SG: Tiny clean up to remove fields now unused from minimal schema [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1287379 (https://phabricator.wikimedia.org/T418885) [11:22:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:25:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:26:42] (03PS1) 10Lars SG: Remove separate endowment and foundation last donation date fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287381 (https://phabricator.wikimedia.org/T418885) [11:30:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [17.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:46:44] (03PS1) 10Lars SG: Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) [11:47:52] (03CR) 10CI reject: [V:04-1] Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [11:48:47] (03CR) 10CI reject: [V:04-1] Remove separate endowment and foundation last donation date fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287381 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [11:50:30] (03PS2) 10Lars SG: Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) [11:52:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:55:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [19.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:02:03] (03PS2) 10Lars SG: Remove separate endowment and foundation last donation date fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287381 (https://phabricator.wikimedia.org/T418885) [12:02:03] (03PS3) 10Lars SG: Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) [12:13:05] (03CR) 10CI reject: [V:04-1] Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [12:43:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [12:48:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [12:49:50] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, and 3 others: Ability to resend emails Civi has already generated - https://phabricator.wikimedia.org/T389693#11921448 (10Lars) [[ https://github.com/civicrm/c... [13:02:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:03:08] (03CR) 10Jgleeson: [C:03+2] "This LGTM. I looked over the silverpop export to see if it impacted things there (and the mediawiki donor export), and it looks like we on" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287381 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [13:09:09] (03PS6) 10Jgleeson: Capture payment_service_id from Gravy responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1275495 (https://phabricator.wikimedia.org/T422416) [13:11:38] (03CR) 10Jgleeson: [C:03+2] "I missed these! ha." [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1287379 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [13:12:24] (03CR) 10Jgleeson: [C:03+2] Update php-queue and predis dependencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283843 (https://phabricator.wikimedia.org/T425578) (owner: 10Ejegg) [13:13:02] (03Merged) 10jenkins-bot: Update php-queue and predis dependencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283843 (https://phabricator.wikimedia.org/T425578) (owner: 10Ejegg) [13:13:22] (03Merged) 10jenkins-bot: Tiny clean up to remove fields now unused from minimal schema [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1287379 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [13:15:07] (03PS3) 10Ejegg: Update psr/cache to 3.0 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286425 (https://phabricator.wikimedia.org/T425578) [13:19:34] (03CR) 10Jgleeson: [C:03+2] "LGTM. I'm guessing we can merge this now after tonight's decision to give up upgrading the addshore library dependency." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286425 (https://phabricator.wikimedia.org/T425578) (owner: 10Ejegg) [13:19:47] (03PS3) 10Ejegg: Update psr/log to 3.0, remove Amazon Pay 1.0 SDK [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286960 (https://phabricator.wikimedia.org/T424497) [13:20:19] (03CR) 10Jgleeson: [C:03+2] Update psr/log to 3.0, remove Amazon Pay 1.0 SDK [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286960 (https://phabricator.wikimedia.org/T424497) (owner: 10Ejegg) [13:20:27] (03Merged) 10jenkins-bot: Update psr/cache to 3.0 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286425 (https://phabricator.wikimedia.org/T425578) (owner: 10Ejegg) [13:20:53] (03Merged) 10jenkins-bot: Update psr/log to 3.0, remove Amazon Pay 1.0 SDK [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286960 (https://phabricator.wikimedia.org/T424497) (owner: 10Ejegg) [13:21:51] thanks jgleeson [13:21:55] np [13:24:05] (03Merged) 10jenkins-bot: Remove separate endowment and foundation last donation date fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287381 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [13:43:17] (03PS5) 10Lars SG: Overwrite name when both existing and incoming email location types are low confidence [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281812 (https://phabricator.wikimedia.org/T425196) [13:53:19] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:53:39] (03CR) 10CI reject: [V:04-1] Extract queuing of audit message [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:53:51] (03PS6) 10Eileen: Extract queuing of audit message [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) [13:53:51] (03PS9) 10Eileen: Fix assumption that the batch is initiated before the date is added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281754 (https://phabricator.wikimedia.org/T419044) [13:53:52] (03PS5) 10Eileen: Move setting Campaign to somewhere less opinionated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281995 (https://phabricator.wikimedia.org/T419044) [13:53:52] (03PS5) 10Eileen: Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) [13:53:53] (03PS3) 10Eileen: Extract contact [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284581 (https://phabricator.wikimedia.org/T419044) [13:53:54] (03PS3) 10Eileen: Extract ContributionRecur [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284582 (https://phabricator.wikimedia.org/T419044) [13:53:58] (03PS3) 10Eileen: Extract saveContribution, queueContributionTracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284583 (https://phabricator.wikimedia.org/T419044) [13:54:02] (03PS3) 10Eileen: Namespace exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284584 [13:55:26] (03CR) 10Jgleeson: [C:03+2] Fix assumption that the batch is initiated before the date is added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281754 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:56:05] (03CR) 10Jgleeson: [C:03+2] Move setting Campaign to somewhere less opinionated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281995 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:56:11] FIRING: NodeDown: Node fran2001 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:56:29] (03CR) 10Jgleeson: [C:03+2] Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:57:29] (03CR) 10Jgleeson: [C:03+2] Extract contact [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284581 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:57:48] (03CR) 10Jgleeson: [C:03+2] Extract ContributionRecur [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284582 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:58:27] (03CR) 10Jgleeson: "Nice refactoring!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284583 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:58:42] (03CR) 10Jgleeson: [C:03+2] Namespace exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284584 (owner: 10Eileen) [13:58:47] FIRING: HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [13:59:47] FIRING: HAProxyBackendDown: HAProxy backend minio_backend on fransw2002.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown [14:01:11] FIRING: [7x] NodeDown: Node franio2001 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:01:11] FIRING: NodeDown: Node fran2001 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran2001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:02:10] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on fransc2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:02:36] FIRING: StrongswanFailure: Strongswan is not fully established on frban2002.frack.codfw.wmnet:9100 for peerhost fran2001 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [14:02:47] FIRING: HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [14:03:42] FIRING: MinIONodesOffline: Instance franio2004.frack.codfw.wmnet:9000 has 3 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [14:03:47] FIRING: [2x] HAProxyExporterDown: HAProxy on fransw2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [14:12:57] FIRING: MariadbNoReplicas: No MariaDB replicas connected to : frdb1007.frack.eqiad.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [14:19:08] (03Merged) 10jenkins-bot: Extract queuing of audit message [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:19:12] (03Merged) 10jenkins-bot: Fix assumption that the batch is initiated before the date is added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281754 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:23:06] (03Merged) 10jenkins-bot: Move setting Campaign to somewhere less opinionated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281995 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:23:08] (03Merged) 10jenkins-bot: Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:23:11] (03Merged) 10jenkins-bot: Extract contact [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284581 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:45:51] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:55:51] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:56:21] (03CR) 10Jgleeson: [C:03+2] Extract saveContribution, queueContributionTracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284583 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:04:51] FIRING: [3x] InboundInterfaceErrors: Inbound errors on interface pfw1-codfw:reth0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Inbound/outbound_interface_errors - https://alerts.wikimedia.org/?q=alertname%3DInboundInterfaceErrors [15:05:51] RESOLVED: [3x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:09:43] (03CR) 10Ejegg: Add Chariot audit processor (034 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:10:40] (03CR) 10Ejegg: "Fixed in I0b1f9906a79a153cf94ed4d93611af57c32cde9c" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:19:44] (03Merged) 10jenkins-bot: Extract saveContribution, queueContributionTracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284583 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:19:46] (03Merged) 10jenkins-bot: Namespace exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284584 (owner: 10Eileen) [15:45:42] (03PS1) 10Jgleeson: Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 [15:46:12] (03CR) 10CI reject: [V:04-1] Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 (owner: 10Jgleeson) [15:47:59] (03PS2) 10Jgleeson: Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 [15:48:27] (03CR) 10CI reject: [V:04-1] Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 (owner: 10Jgleeson) [15:51:30] (03PS3) 10Jgleeson: Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 [15:51:46] I swear I put that WIP to avoid the irc spam [15:52:04] oh right - git review over a WIP patch un-WIPs it [15:52:44] oh no it doesn't - ok it's a me problem then [15:54:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [142.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:55:25] (03PS3) 10Jgleeson: Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 [15:55:42] cstone: that should quiet down our monitoring IPN failmails ^ [15:55:54] It's nice of Gr4vy to turn them on :D [15:56:05] thanks jgleeson ! [15:56:07] i know right.... [15:56:32] (03CR) 10Cstone: [C:03+2] "thanks!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 (owner: 10Jgleeson) [15:57:05] (03Merged) 10jenkins-bot: Handle new Gravy Monitoring Incident IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287437 (owner: 10Jgleeson) [15:59:02] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287438 [15:59:26] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287438 (owner: 10Jgleeson) [15:59:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [141.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:02:39] 16:59:22 In GitDownloader.php line 155: [16:02:39] 16:59:22 [16:02:39] 16:59:22 The .git directory is missing from /src/vendor/coderkungfu/php-queue, see h [16:02:39] 16:59:22 ttps://getcomposer.org/commit-deps for more information [16:02:39] 16:59:22 [16:02:40] great [16:28:40] (03CR) 10Wfan: [C:03+2] Accept landing_page as a top-level c_t field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1279467 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [16:31:36] (03PS1) 10Ejegg: Update predis, psr-cache, psr-log remove amzn SDK [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1287442 [16:31:39] (03CR) 10Ejegg: [C:03+2] Update predis, psr-cache, psr-log remove amzn SDK [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1287442 (owner: 10Ejegg) [16:32:17] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287444 [16:32:20] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287444 (owner: 10Ejegg) [16:37:33] FIRING: RsyslogBacklog: 8 ryslog files queued up on fransc2001.frack.codfw.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [16:38:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [16:42:33] RESOLVED: RsyslogBacklog: 5 ryslog files queued up on fransc2001.frack.codfw.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [16:45:28] oh huh smashpig/vendor patch doesn't seem to be in gate+submit [16:45:38] will just submit it [16:45:42] (03CR) 10Ejegg: [V:03+2 C:03+2] Update predis, psr-cache, psr-log remove amzn SDK [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1287442 (owner: 10Ejegg) [16:45:55] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287444 (owner: 10Ejegg) [16:46:20] ok jgleeson I'll push that out. Will be watching the logs for a few min at least [16:46:36] oh right [16:46:39] thanks ejegg [16:48:18] !log standalone (IPN listener) SmashPig upgraded from bb833986 to a2a3a015 [16:48:18] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [16:49:11] IPN listener logs seem fine so far [16:49:27] have seen stuff from most active services [16:49:58] oh the job runner logs would be good to look at too [16:50:06] nice [16:50:11] gonna grab food [16:52:36] (03Merged) 10jenkins-bot: Accept landing_page as a top-level c_t field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1279467 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [16:53:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [96.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:53:54] thanks for the CR wfan [16:55:06] np [17:07:41] (03PS1) 10Ejegg: Change max sync refunds to setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287451 (https://phabricator.wikimedia.org/T421277) [17:08:03] fr-tech that's the setting that should let DR start using the refunds form ^^^ [17:08:08] gotta go help with lunch [17:23:40] 06Fundraising-Backlog, 10FR-Donor-portal: [Donor Portal] Update subject line in 'Problems logging in?' dialog box (hardcoded) - https://phabricator.wikimedia.org/T426345 (10KHill-WMF) 03NEW [17:31:22] (03CR) 10Wfan: [C:03+2] Change max sync refunds to setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287451 (https://phabricator.wikimedia.org/T421277) (owner: 10Ejegg) [17:32:09] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [90.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [17:33:37] 10fundraising-tech-ops, 10FR-Tech-Analytics: Configure Fundraising eqiad analytics cluster - https://phabricator.wikimedia.org/T398317#11922925 (10Jgreen) 05Open→03Resolved p:05Triage→03Medium [17:54:11] (03Merged) 10jenkins-bot: Change max sync refunds to setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287451 (https://phabricator.wikimedia.org/T421277) (owner: 10Ejegg) [18:10:47] thanks again wfan [18:14:12] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923024 (10AStein-WMF) so... [18:18:11] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923037 (10AStein-WMF) @K... [18:20:59] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923042 (10AStein-WMF) al... [18:22:51] oh dang, omnimail is still pointing at psr/log 1.0 [18:24:08] I bet it could allow up to 3.0 w/o code changes [18:24:38] ok, so not a thing I can fix for this deploy [18:26:54] (03PS1) 10Ejegg: Update SmashPig and other dependencies [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287458 [18:26:56] (03CR) 10Ejegg: [C:03+2] Update SmashPig and other dependencies [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287458 (owner: 10Ejegg) [18:27:26] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287459 [18:28:15] (03Abandoned) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1287438 (owner: 10Jgleeson) [18:29:23] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2526Q4 maintenance window - https://phabricator.wikimedia.org/T364186#11923046 (10Dwisehaupt) [18:31:00] oh huh, omnimail/omnimail already allows psr/log 3.0 from what I can see... I wonder what version we're on? [18:37:01] ohhhh we're using eileen's repo for omnimail/omnimail [18:37:25] let's see if there's anything incompatible there [18:40:05] ah, and it's also pegged to a specific commit :( [18:40:34] well, let's just try unlocking that completely :) [18:43:07] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2526Q4 maintenance window - https://phabricator.wikimedia.org/T364186#11923076 (10Dwisehaupt) [18:45:24] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923086 (10SHust) @AStein... [18:46:17] oh nice, no big omnimail changes [18:47:54] (03PS1) 10Ejegg: Update smash-pig, remove amzn sdk [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287463 [18:48:11] so let's see what the whole vendor diff is loke [18:49:37] (03Merged) 10jenkins-bot: Update SmashPig and other dependencies [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287458 (owner: 10Ejegg) [18:50:57] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923100 (10Lars) @AStein-... [18:51:42] (03PS1) 10Ejegg: Update SmashPig, predis, psr/cache, psr/log [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287464 [18:52:14] ok, let's see how many tests pass locally for that [18:53:10] FIRING: PuppetFailure: Puppet has failed on fran2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:53:31] looks like a hefty commit but most of it is predis/predis update to 3.x [18:55:16] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923106 (10SHust) @Lars,... [18:56:11] FIRING: [3x] NodeDown: Node franio2004 has been down for long. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:57:42] FIRING: [4x] MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [19:00:57] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287459 (owner: 10Ejegg) [19:01:53] ok, that CRM deploy has enough refactors that I want to stop queues to push it out then run a few slow-starts [19:02:01] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287459 (owner: 10Ejegg) [19:02:05] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1287459/-1..1 [19:04:51] !log disabled fundraising scheduled jobs for CiviCRM deployment [19:04:52] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [19:05:06] FIRING: [3x] InboundInterfaceErrors: Inbound errors on interface pfw1-codfw:reth0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Inbound/outbound_interface_errors - https://alerts.wikimedia.org/?q=alertname%3DInboundInterfaceErrors [19:06:11] FIRING: NodeDown: Node payments1012 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1012 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:06:56] !log fundraising civicrm upgraded from 950908ec to e25fa223 [19:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [19:07:45] ok, reconciled managed entities [19:07:50] let's see if the new setting is there [19:08:11] hmm, not that I see [19:08:14] maybe flush too? [19:09:13] huh, still not at https://civicrm.wikimedia.org/civicrm/admin/setting/wmf-civicrm [19:09:18] did I mess that up? [19:09:58] ohhh i didn't add the settings_pages line [19:10:04] well, it should be in the db anyway [19:10:38] yep, shows up with cv setting:get [19:11:00] ok, now set to 25 [19:11:11] RESOLVED: NodeDown: Node payments1012 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1012 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:11:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [19:15:18] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make payments_fraud tables available in SK and add to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11923141 (10Lars) [19:16:12] (03Abandoned) 10Lars SG: WIP: Roll up of upstream cv changes since Feb 2026 (4ae6191 → 41e34b6) [wikimedia/fundraising/cv] - 10https://gerrit.wikimedia.org/r/1275546 (https://phabricator.wikimedia.org/T423618) (owner: 10Lars SG) [19:22:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Updating the Contribution History on the Contact Summary to reflect the current FY totals - https://phabricator.wikimedia.org/T424937#11923165 (10Lars) a:03Lars [19:23:33] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Batch refund action for unintended recurring contribution - https://phabricator.wikimedia.org/T426350 (10Ejegg) 03NEW [19:24:27] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Fraud: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11923179 (10Ejegg) OK, this is ready... [19:28:10] RESOLVED: PuppetFailure: Puppet has failed on fran2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:29:31] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#11923186 (10Dwisehaupt) Alters done: ` MariaDB [civicrm]> ALTER TABLE wmf_donor DROP INDEX entity_total... [19:29:51] !log re-enabled fundraising scheduled jobs [19:29:52] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [19:36:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 for >30m - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [19:45:10] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923216 (10AStein-WMF) Gr... [19:46:40] eileen: would it be OK to point omnimail/omnimail back at the main repo? looks like they've merged all that's in your repo, plus a couple patches to allow newer psr/log [19:47:29] here's a lib update patch that gets us up to date with predis, psr/log, and psr/cache: https://gerrit.wikimedia.org/r/1287463 [19:49:43] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923220 (10Lars) @AStein-... [20:06:11] FIRING: NodeDown: Node payments1012 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1012 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:06:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [20:11:11] FIRING: [4x] NodeDown: Node franio2004 has been down for long. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:15:44] ah dang [20:15:59] I should have allowed more versions of psr/log in smashpig [20:16:16] MW 1.43 is pegged to psr/log 1.1.4 [20:18:44] (03PS1) 10Ejegg: Allow more versions of psr/log [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287476 [20:18:47] (03CR) 10Ejegg: [C:03+2] Allow more versions of psr/log [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287476 (owner: 10Ejegg) [20:19:05] one more tag... [20:19:20] (03Merged) 10jenkins-bot: Allow more versions of psr/log [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1287476 (owner: 10Ejegg) [20:19:51] RESOLVED: [3x] InboundInterfaceErrors: Inbound errors on interface pfw1-codfw:reth0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Inbound/outbound_interface_errors - https://alerts.wikimedia.org/?q=alertname%3DInboundInterfaceErrors [20:25:45] ok, that will actually make the CRM SmashPig update slightly simpler [20:31:31] (03PS2) 10Ejegg: Update smash-pig, remove amzn sdk [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287463 [20:32:30] that seems safe enough [20:32:46] (03CR) 10Ejegg: [C:03+2] Update smash-pig, remove amzn sdk [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287463 (owner: 10Ejegg) [20:36:06] hmm, so we could move the gravy sdk to the 'suggests' part of the SmashPig composer.json, then require it directly under civicrm and our MW 1.43 branch [20:36:29] that would let DonationInterface install happily on 1.47 [20:36:43] and give us another 6 months to figure out what to do with that jwt version [20:40:12] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923364 (10AStein-WMF) Do... [20:41:17] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923366 (10Lars) Great, t... [20:46:11] RESOLVED: NodeDown: Node frmx2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmx2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:46:11] FIRING: [4x] NodeDown: Node franio2004 has been down for long. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:47:58] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11923379 (10AStein-WMF) yo... [20:51:11] RESOLVED: NodeDown: Node frmx2002 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmx2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:55:02] (03Merged) 10jenkins-bot: Update smash-pig, remove amzn sdk [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287463 (owner: 10Ejegg) [21:06:10] (03CR) 10Eileen: [C:03+2] Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [21:09:08] ejegg: sample one [21:09:08] {"gateway":"adyen","gateway_txn_id":"PNXD2V2H3MJDG969","backend_processor":"adyen","backend_txn_id":"PNXD2V2H3MJDG969","audit_file_gateway":"adyen","gateway_account":"WikimediaDonations","invoice_id":"206873633.24","settlement_batch_reference":"1182","exchange_rate":"0.006268656716418","settled_date":1778633252,"contribution_tracking_id":206873633,"payment_method":"cc","payment_submethod":"jcb","date":1778633252,"bac [21:09:08] kend_processor_txn_id":"PNXD2V2H3MJDG969","type":"donation","modification_reference":"LKMJGFC7Q9PPZCB9","capture_id":"LKMJGFC7Q9PPZCB9","auth_id":"PNXD2V2H3MJDG969","original_currency":"JPY","currency":"JPY","gross":"335","original_total_amount":"335","fee":"11","original_fee_amount":"11","settled_gross":"2.03","settled_net_amount":"2.03","settled_currency":"USD","settled_fee_amount":"-0.07","original_net_amount":"32 [21:09:08] 4","settled_total_amount":"2.10"} [21:09:08] https://civicrm.wikimedia.org/civicrm/contact/view/contribution?reset=1&action=view&id=143145375 [21:11:33] FIRING: RsyslogBacklog: 19 ryslog files queued up on franio2004.frack.codfw.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [21:12:42] RESOLVED: [4x] MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [21:16:33] RESOLVED: RsyslogBacklog: 19 ryslog files queued up on franio2004.frack.codfw.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [21:19:41] FIRING: [2x] NodeDown: Node frqueue2004 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:23:07] (03PS2) 10Lars SG: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) [21:23:07] (03PS1) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [21:23:35] looking eileen [21:24:41] RESOLVED: [2x] NodeDown: Node franio2004 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:26:31] oh weird, gateway_txn_id is right but backend_processor_txn_id is wrong? [21:28:00] yep, that's what it sent from the recurring job [21:28:03] so why? [21:28:14] and will the recent refactor help? [21:35:50] (03Merged) 10jenkins-bot: Remove disabled custom fields for accounting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287382 (https://phabricator.wikimedia.org/T418885) (owner: 10Lars SG) [21:36:00] (03PS2) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [21:36:06] eileen: I think the deploy at 19:06 UTC actually fixed those! [21:36:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Clean up emails where contact has two of the same type and address after backfilling location types - https://phabricator.wikimedia.org/T425190#11923497 (10Lars) a:03Lars [21:40:42] (03PS1) 10Lars SG: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287489 [21:41:26] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287489 (owner: 10Lars SG) [21:42:04] larssandergreen: sorry, I need to do another vendor update [21:42:15] doing that now [21:42:29] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [21:42:47] this one updates predis from 1.x to 3.x, so I think I want to pause jobs again [21:44:00] ejegg: just noticed that was in there, was about to do it [21:45:11] lemme just re-use this change-id [21:45:18] (03PS2) 10Ejegg: Update SmashPig, predis, psr/cache [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287464 [21:45:47] ok that looks right [21:45:53] (03CR) 10Ejegg: [C:03+2] Update SmashPig, predis, psr/cache [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287464 (owner: 10Ejegg) [21:46:32] (03PS2) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287489 (owner: 10Lars SG) [21:46:55] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287489 (owner: 10Lars SG) [21:55:06] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [22:02:58] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [22:03:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frpm1002.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:05:15] ejegg: cool - I will see where those ids are at over next few days [22:06:38] (03Merged) 10jenkins-bot: Update SmashPig, predis, psr/cache [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1287464 (owner: 10Ejegg) [22:06:40] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287489 (owner: 10Lars SG) [22:08:54] ejegg: i'll leave you to deploy that one [22:09:23] !log fundraising scheduled jobs disabled for Civi update [22:09:24] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [22:10:23] oh I'm going to disable the amazon job runner too [22:10:41] I mean not turn it back on [22:11:00] !log fundraising civicrm upgraded from e25fa223 to 6d8ce7a3 [22:11:01] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [22:11:12] ok, lemme run some test message consumes [22:12:27] !log cv upgraded from f19e0961 to b8a8dd6a [22:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [22:13:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frpm1002.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:14:06] ok, seems to be getting things from predis just fine [22:14:23] will turn em all back on, minus that amazon thing which would just fail anyway [22:18:20] oho, those adyen capture job runners seem to be still in use, to capture autorescued auths [22:18:31] no need to have 3 of them now though [22:19:16] larssandergreen: ok, that's deployed [22:19:33] i did a cv flush but no other upgrade-type commands [22:21:07] I think we need WMFConfig.SyncCustomFields, right? [22:21:36] will run it [22:22:31] !log fundraising scheduled jobs re-enabled [22:22:32] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [22:22:40] ok, syncCustomFields is done [22:23:30] and I don't see finance only in the contribution fields [22:24:08] oh wait, it's still there in the admin screen: https://civicrm.wikimedia.org/civicrm/admin/custom/group/fields#/?gid=16 [22:24:15] just disabled [22:25:02] larssandergreen: were you expecting that to do the field deletions and table alter? [22:25:20] oops, gotta help with dinner [22:26:05] (03PS4) 10Lars SG: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) [22:28:04] ejegg: nope, the fields will need to be deleted manually (after the trigger update) [22:28:22] (03PS1) 10Ejegg: Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 [22:28:24] ah right [22:29:38] (03CR) 10CI reject: [V:04-1] Update SmashPig, predis/predis and psr/cache [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1287490 (owner: 10Ejegg) [22:30:29] (03PS5) 10Lars SG: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) [22:31:18] eileen or ejegg: could one of you review the trigger changes above? [22:31:40] looking [22:32:13] dwisehaupt: I'm signing off, but the trigger updates are above and if you want to deploy them once they've been reviewed, feel free (or wait until tomorrow is fine with me too) [22:32:16] those spacing changes might be because I tried to prevent trailing spaces recently-ish [22:32:41] larssandergreen: ok, thanks. have a good night. [22:33:12] eileen: oh yes, I forgot about that [22:34:23] (03CR) 10Eileen: [C:03+2] Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) (owner: 10Lars SG) [22:34:49] looks fine - they consolidated last_donation_date I see - [22:57:10] (03Merged) 10jenkins-bot: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) (owner: 10Lars SG) [23:00:30] (03CR) 10Krinkle: Move ext.centralNotice.startUp from onBeforePageDisplay to onSiteNoticeAfter (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1285504 (https://phabricator.wikimedia.org/T422082) (owner: 10Krinkle) [23:08:44] (03CR) 10Wfan: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [23:12:20] (03PS3) 10Wfan: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [23:13:10] (03PS3) 10Lars SG: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) [23:33:35] (03CR) 10CI reject: [V:04-1] Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [23:34:43] (03CR) 10CI reject: [V:04-1] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [23:40:40] (03PS4) 10Wfan: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [23:57:46] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1287500