[00:09:17] !log civicrm upgraded from fcbbf763 to cae2ab0e [00:09:18] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [00:12:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Recurring-Donations, 13Patch-For-Review: Some PayPal recurrings recorded with bad conversion rate - https://phabricator.wikimedia.org/T426098#11943316 (10Cstone) We deployed the tiny fix to stop new ones from happening today, I will keep an eye on i... [01:58:37] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290146 [02:00:07] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1290149 [02:00:53] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1290149 (owner: 10Eileen) [02:01:13] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290146 (owner: 10Eileen) [02:22:39] (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1290149 (owner: 10Eileen) [02:23:19] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Look into SQL options to avoid long-running queries from Civi UI - https://phabricator.wikimedia.org/T426927 (10Ejegg) 03NEW [02:23:45] (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.2.4.10 => v1.2.4.11) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290146 (owner: 10Eileen) [02:29:07] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1290160 [04:02:57] (03PS5) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [04:06:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:11:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:11:58] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1290160 (owner: 10Eileen) [04:13:14] !log civicrm upgraded from cae2ab0e to dbafc0b4 [04:13:15] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [04:56:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:06:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:09:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:14:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:18:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:23:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:30:21] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:31:29] (03PS6) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [05:35:21] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:52:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:53:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [06:02:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [06:12:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:12:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:12:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:17:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:17:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:17:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:29:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [9.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:29:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [9.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:29:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [9.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:42:41] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [06:44:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:44:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:44:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:53:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [07:56:23] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_45) - 10https://gerrit.wikimedia.org/r/1290377 (owner: 10L10n-bot) [07:58:36] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290594 (owner: 10L10n-bot) [10:16:43] (03PS7) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [10:22:54] (03PS8) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [10:24:29] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07fr-current-sprint: Integrate Chariot to Civi for automated payment info updates - https://phabricator.wikimedia.org/T419044#11944157 (10Eileenmcnaughton) @MDemosWMF I have had an initial go at pulling these in onto staging - you can see them here... [10:42:41] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [10:47:23] (03CR) 10CI reject: [V:04-1] Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [11:22:27] (03CR) 10Lars SG: [C:03+2] Update recentmenu [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290074 (owner: 10Eileen) [11:23:56] (03CR) 10Lars SG: [C:03+2] Update contact layout editor extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290073 (owner: 10Eileen) [11:24:58] (03CR) 10Lars SG: [C:03+2] Update civitoken [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290072 (owner: 10Eileen) [11:29:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:34:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:48:27] (03Merged) 10jenkins-bot: Update civitoken [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290072 (owner: 10Eileen) [11:49:16] (03Merged) 10jenkins-bot: Update contact layout editor extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290073 (owner: 10Eileen) [11:50:42] (03Merged) 10jenkins-bot: Update recentmenu [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290074 (owner: 10Eileen) [12:19:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1017 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:24:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1017 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:25:30] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824#11944579 (10Jgreen) >>! In T426824#11942813, @Jhancock.wm wrote: > i can get this one in the morning if Jeff or Dallas is around and want to coordinate. @Jhancock.wm I'... [12:34:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:44:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1017 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:49:10] RESOLVED: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:55:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:00:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:20:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Recurring-Donations, 07payments-orchestration: Recurring bank charges fail with "Payment method has not been successfully stored." - https://phabricator.wikimedia.org/T426963 (10Ejegg) 03NEW [13:25:35] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [14:21:11] FIRING: NodeDown: Node frdata1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdata1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:22:41] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [14:26:11] RESOLVED: NodeDown: Node frdata1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdata1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:27:41] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [14:47:51] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:xe-0/2/1 (Core: fasw2-c8a-codfw:xe-0/0/47 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:48:02] FIRING: SwitchCoreInterfaceDown: Switch core interface down - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:52:51] RESOLVED: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:xe-0/2/1 (Core: fasw2-c8a-codfw:xe-0/0/47 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:52:56] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:53:51] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:xe-0/2/1 (Core: fasw2-c8a-codfw:xe-0/0/47 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:54:02] FIRING: SwitchCoreInterfaceDown: Switch core interface down - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:58:51] RESOLVED: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:xe-0/2/1 (Core: fasw2-c8a-codfw:xe-0/0/47 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:59:51] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:ge-0/2/1 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:03:31] FIRING: Emergency syslog message: Alert for device pfw1-codfw.wikimedia.org - Emergency syslog message - https://alerts.wikimedia.org/?q=alertname%3DEmergency+syslog+message [15:04:51] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:ge-0/2/1 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:13:31] RESOLVED: Emergency syslog message: Device pfw1-codfw.wikimedia.org recovered from Emergency syslog message - https://alerts.wikimedia.org/?q=alertname%3DEmergency+syslog+message [15:13:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:14:38] (03CR) 10Jgleeson: [C:03+2] "This looks safe as it's already happening earlier here https://github.com/wikimedia/wikimedia-fundraising-crm/blob/e9e0c8f3b300c8589151116" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290092 (owner: 10Eileen) [15:15:41] (03CR) 10Jgleeson: "I'll leave this one for someone more familiar with Chariot" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1290079 (owner: 10Eileen) [15:18:44] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290041 (owner: 10Ejegg) [15:25:27] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824#11945503 (10Jhancock.wm) a:03Jhancock.wm got it fixed. fiber patch is good. optic in fasw2-c8a-codfw is good. it was the optic in the pfw sending bad light. cleaning he... [15:48:16] (03Merged) 10jenkins-bot: Move away from vague 'gross', 'fee', 'currency' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290092 (owner: 10Eileen) [15:49:49] (03Merged) 10jenkins-bot: Fix undefined var in refund QC [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290041 (owner: 10Ejegg) [15:57:40] (03PS1) 10Ejegg: Add refund activity from new refund form [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290825 (https://phabricator.wikimedia.org/T421277) [16:00:52] (03PS2) 10Ejegg: Add refund activity from new refund form [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290825 (https://phabricator.wikimedia.org/T421277) [16:18:11] FIRING: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:19:30] 10fundraising-tech-ops: Fundraising mariadb database server/service tuning - https://phabricator.wikimedia.org/T423950#11945720 (10Jgreen) frdb1006 (staging) has BIOS adjustment and mariadb tuning applied [16:23:11] RESOLVED: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:27:11] FIRING: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:42:11] RESOLVED: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:23:10] 14Fundraising Sprint - GNU England Shaker dresser, 14Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11945817 (10MBeat33) Demo-ed thi... [17:36:31] 14Fundraising Sprint - GNU England Shaker dresser, 14Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11945876 (10MBeat33) 05Resolved→0... [18:14:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:24:54] (03CR) 10Wfan: [C:03+2] Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [18:27:35] (03CR) 10Wfan: [C:03+2] Overwrite name when both existing and incoming email location types are low confidence [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281812 (https://phabricator.wikimedia.org/T425196) (owner: 10Lars SG) [18:27:42] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [18:29:38] (03PS1) 10Damilare Adedoyin: DonorPortal: Add text for legacy paypal to explain why they currently cannnot be managed on the DonorPortal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) [18:30:40] RESOLVED: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:39:10] FIRING: PuppetFailure: Puppet has failed on franio2004.frack.codfw.wmnet:9100 - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:47:21] (03Merged) 10jenkins-bot: Clean up emails where a contact has two of the same type and address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287487 (https://phabricator.wikimedia.org/T425190) (owner: 10Lars SG) [18:51:35] (03PS1) 10Damilare Adedoyin: Donor Portal: Remove duplicated unit tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290880 [18:51:47] (03CR) 10CI reject: [V:04-1] Donor Portal: Remove duplicated unit tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290880 (owner: 10Damilare Adedoyin) [18:52:00] (03PS2) 10Damilare Adedoyin: DonorPortal: Add text for legacy paypal to explain why they currently cannnot be managed on the DonorPortal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) [18:52:02] (03Merged) 10jenkins-bot: Overwrite name when both existing and incoming email location types are low confidence [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281812 (https://phabricator.wikimedia.org/T425196) (owner: 10Lars SG) [18:55:19] (03PS3) 10Damilare Adedoyin: DonorPortal: Add text for legacy paypal to explain why they currently cannnot be managed on the DonorPortal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) [18:55:19] (03PS2) 10Damilare Adedoyin: Donor Portal: Remove duplicated unit tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290880 [18:55:34] 14Fundraising Sprint - GNU England Shaker dresser, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition, 13Patch-For-Review: Add SFTP upload support to MediaWiki donor export job - https://phabricator.wikimedia.org/T421772#11946127 (10jgleeson) 05Open→03Resolved [19:00:02] (03PS3) 10Damilare Adedoyin: Donor Portal: Remove duplicated unit tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290880 [19:09:10] FIRING: [3x] PuppetFailure: Puppet has failed on franio1004.frack.eqiad.wmnet:9100 - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:14:10] FIRING: [3x] PuppetFailure: Puppet has failed on franio1004.frack.eqiad.wmnet:9100 - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:19:10] RESOLVED: [3x] PuppetFailure: Puppet has failed on franio1004.frack.eqiad.wmnet:9100 - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:03:03] (03CR) 10CI reject: [V:04-1] build: Updating npm dependencies [extensions/FundraisingTranslateWorkflow] (REL1_43) - 10https://gerrit.wikimedia.org/r/1157359 (owner: 10Libraryupgrader) [20:18:22] (03PS1) 10Umherirrender: tests: Remove extra TestCase::returnValue() [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290909 [20:29:51] (03CR) 10Ejegg: "The code change looks fine, but it's still a lot of text for that space. This might be a little better: "Processed via PayPal. Changes mus" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290874 (https://phabricator.wikimedia.org/T421962) (owner: 10Damilare Adedoyin) [20:33:08] (03CR) 10Ejegg: [C:03+2] tests: Remove extra TestCase::returnValue() [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290909 (owner: 10Umherirrender) [20:36:00] (03Merged) 10jenkins-bot: tests: Remove extra TestCase::returnValue() [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290909 (owner: 10Umherirrender) [20:38:39] (03PS8) 10Jgleeson: Capture payment_service_id from Gravy responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1275495 (https://phabricator.wikimedia.org/T422416) [20:39:22] (03CR) 10Ejegg: [C:03+2] "This looks good! Tested with the DonationInterface patch and I see the service ID getting sent to the pending queue. Sorry for the long de" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1275495 (https://phabricator.wikimedia.org/T422416) (owner: 10Jgleeson) [20:39:59] (03Merged) 10jenkins-bot: Capture payment_service_id from Gravy responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1275495 (https://phabricator.wikimedia.org/T422416) (owner: 10Jgleeson) [20:40:08] thank you ejegg ! [21:14:03] (03CR) 10Eileen: "This 'found field' list is a bit of an oddball - it was my friend's idea but it's useful in the short-term. Basically it's a list of all t" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1290079 (owner: 10Eileen) [22:12:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:17:12] (03CR) 10Cstone: [C:03+2] Note another 'found' field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1290079 (owner: 10Eileen) [22:17:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:17:52] (03Merged) 10jenkins-bot: Note another 'found' field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1290079 (owner: 10Eileen) [22:18:45] (03CR) 10Eileen: [V:03+2] Composer update - fairly tame [wikimedia/fundraising/cv] - 10https://gerrit.wikimedia.org/r/1285734 (owner: 10Eileen) [22:19:54] (03PS2) 10Eileen: Fix notice [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284445 [22:20:58] (03CR) 10CI reject: [V:04-1] Fix notice [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284445 (owner: 10Eileen) [22:21:49] 06Fundraising-Backlog, 10fundraising-tech-ops: Monitor the success percentage for each payment methods - https://phabricator.wikimedia.org/T407927#11946806 (10Dwisehaupt) This is now set up with the alertmanager rules in place. They are in the puppet repo: `modules/prometheus/files/rules/payments.yml` More ru... [22:22:25] (03PS3) 10Eileen: Fix notice [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284445 [22:23:06] (03PS2) 10Eileen: Upgrading wikimedia/omnimail-silverpop (1.31 => 1.32) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283759 [22:27:42] FIRING: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [22:32:26] ^^^ that is tmpreaper getting in the way of the public data export again. [22:32:53] i'm re-running the export and data copy, and looking at shifting it from /tmp to /srv/tmp to keep it from happening again. [22:52:42] RESOLVED: [2x] EndpointResultWarning: Endpoint check for fr-data on frdata1002 is WARNING - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultWarning [23:32:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [23:47:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is high: [2089] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [23:52:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is Critical: [2567] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [23:53:43] i'll ack that since we have the queues stopped. [23:57:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService