[00:43:20] (03PS1) 10Eileen: Chariot audit - Declare another newly found field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289450 (https://phabricator.wikimedia.org/T419044) [00:43:20] (03PS1) 10Eileen: Chariot Audit - fix notice [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289451 (https://phabricator.wikimedia.org/T419044) [01:22:24] (03CR) 10Cstone: [C:03+2] Notice fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289438 (owner: 10Eileen) [01:22:39] (03CR) 10Cstone: [C:03+2] Chariot audit - Declare another newly found field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289450 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [01:22:55] (03CR) 10Cstone: [C:03+2] Chariot Audit - fix notice [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289451 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [01:23:17] (03Merged) 10jenkins-bot: Chariot audit - Declare another newly found field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289450 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [01:23:32] (03Merged) 10jenkins-bot: Chariot Audit - fix notice [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289451 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [01:25:49] (03CR) 10Cstone: "yeah I switched it to just check for a normalized currency variable before doing the currency normalization" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [01:26:15] (03PS5) 10Cstone: Only normalize currency if it hasn't been normalized yet [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) [01:30:13] 03Fundraising Sprint: Infinity Pool, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add Searchkit with daily email notifications / dashlet for donations missing required details - https://phabricator.wikimedia.org/T412463#11938890 (10Cstone) Talked to... [01:37:46] (03CR) 10Eileen: "Are these already set in the incoming message? If so maybe getOriginalAmount should start like" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [01:39:09] (03CR) 10Cstone: "incoming as in the normalize method? The bug here is that paypal recurrings are getting normalized twice and double currency exchange rate" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [01:45:04] (03Merged) 10jenkins-bot: Notice fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289438 (owner: 10Eileen) [02:09:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [02:14:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [02:18:46] (03PS1) 10Cstone: Add example paypal payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289457 [03:03:47] (03PS1) 10Eileen: Include matching_gift_organization in csv file [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289461 (https://phabricator.wikimedia.org/T419044) [03:10:26] (03CR) 10Eileen: [C:03+2] Add example paypal payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289457 (owner: 10Cstone) [03:10:40] thanks eileen [03:12:09] eileen: i got really close to fundraise up paypal ftp account today but then the UI would not let me add a new one and the ai help was not being helpful im hoping itll work tomorrow [03:32:19] lol [03:32:39] Evelyn didn't think we have the power.... [03:32:44] but ... maybe? [03:37:46] yeah julianne and i got pretty close [04:23:21] (03PS3) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [04:37:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [04:38:04] (03CR) 10CI reject: [V:04-1] Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [04:42:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:16:01] (03PS4) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [05:16:01] (03PS1) 10Eileen: Extract populateSupportedFieldsFromDocBlock [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289625 (https://phabricator.wikimedia.org/T419044) [05:17:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:22:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:26:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:31:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [05:37:21] (03CR) 10CI reject: [V:04-1] Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [07:12:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [07:13:33] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Using Acoustic API to opt users into SMS program via paymentswiki variant form - https://phabricator.wikimedia.org/T423967#11939277 (10Lars) [07:17:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [07:29:59] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Implement new audience framework in CiviCRM and export to Acoustic - https://phabricator.wikimedia.org/T426818 (10Lars) 03NEW [07:30:37] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Implement new audience framework in CiviCRM and export to Acoustic - https://phabricator.wikimedia.org/T426818#11939319 (10Lars) [07:35:54] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Request for improvement to prospecting system to track moves management - https://phabricator.wikimedia.org/T386961#11939334 (10Lars) [07:40:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [07:45:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [07:48:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [07:54:29] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289822 (owner: 10L10n-bot) [08:26:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [08:31:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [08:33:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:47:02] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824 (10ayounsi) 03NEW p:05Triage→03Medium [08:48:53] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824#11939523 (10ayounsi) [11:20:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1010 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [11:25:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1010 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [11:29:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:32:31] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289822 (owner: 10L10n-bot) [11:39:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:50:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1010 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [11:52:01] 03Fundraising Sprint: Infinity Pool: Sprint H Priorities - https://phabricator.wikimedia.org/T426765#11940250 (10Damilare) 05Open→03Resolved [11:57:36] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 14Fundraising-Backlog-Old, 10MediaWiki-extensions-DonationInterface, and 4 others: Some messages in the Donation extensions are outdated and should be removed - https://phabricator.wikimedia.org/T111677#11940269 (10Pcoombe) @Ejegg You deleted `don... [12:05:42] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10FR-Donor-portal: [Donor Portal] Update subject line in 'Problems logging in?' dialog box (hardcoded) - https://phabricator.wikimedia.org/T426345#11940310 (10Damilare) [12:09:59] (03PS1) 10Damilare Adedoyin: Update Donor Portal log in subject [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289939 [12:10:27] (03PS2) 10Damilare Adedoyin: Update Donor Portal log in subject [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289939 (https://phabricator.wikimedia.org/T426345) [12:10:58] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10FR-Donor-portal, 13Patch-For-Review: [Donor Portal] Update subject line in 'Problems logging in?' dialog box (hardcoded) - https://phabricator.wikimedia.org/T426345#11940335 (10Damilare) a:03Damilare [12:21:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1011 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:26:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1011 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:30:14] (03CR) 10Jgleeson: [C:03+2] "LGTM" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289939 (https://phabricator.wikimedia.org/T426345) (owner: 10Damilare Adedoyin) [12:30:41] thanks jgleeson ! [12:33:43] (03Merged) 10jenkins-bot: Update Donor Portal log in subject [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289939 (https://phabricator.wikimedia.org/T426345) (owner: 10Damilare Adedoyin) [12:34:06] (03CR) 10Jgleeson: [C:03+2] "All this extraction is reminding me of this beauty LOL https://youtu.be/7EmboKQH8lM?t=3549" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289625 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [12:36:07] (03PS7) 10Jgleeson: Capture payment_service_id from Gravy responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1275495 (https://phabricator.wikimedia.org/T422416) [12:36:30] (03PS4) 10Jgleeson: Store and replay payment_service_id for Gravy recurring charges [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1278542 (https://phabricator.wikimedia.org/T422416) [12:36:39] np damilare ! [12:43:38] (03CR) 10Jgleeson: [C:03+2] Remove translated currency names [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288880 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [12:49:30] (03Merged) 10jenkins-bot: Remove translated currency names [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288880 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [12:51:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1011 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:57:18] (03Merged) 10jenkins-bot: Extract populateSupportedFieldsFromDocBlock [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289625 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [13:21:32] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11940725 (10Jgreen) [13:23:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1012 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:24:31] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07Epic, 07fr-combowiki, 07fr-current-sprint: Add Native Payments config API endpoint to payment-wiki - https://phabricator.wikimedia.org/T426858 (10jgleeson) 03NEW [13:25:20] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07Epic, 07fr-combowiki, 07fr-current-sprint: Add Native Payments config API endpoint to payment-wiki - https://phabricator.wikimedia.org/T426858#11940793 (10jgleeson) [13:27:57] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07Epic, 07fr-combowiki, 07fr-current-sprint: Add Native Payments config API endpoint to payments-wiki - https://phabricator.wikimedia.org/T426858#11940821 (10jgleeson) [13:28:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1012 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:34:51] FIRING: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [13:35:46] y'all chose an amazing picture for that sprint name [13:36:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb1007.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb1007.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [13:39:42] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 14Fundraising-Backlog-Old, 10MediaWiki-extensions-DonationInterface, and 3 others: Some messages in the Donation extensions are outdated and should be removed - https://phabricator.wikimedia.org/T111677#11940876 (10Ejegg) Sorry, @Pcoombe, I includ... [13:39:51] RESOLVED: InterfaceOpticLowPower: Too low optic power on - fasw2-c8a-codfw:xe-0/0/47 (Core: pfw1-codfw:xe-0/2/1 {#11519}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=fasw2-c8a-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [13:48:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1012 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:52:36] (03PS1) 10Ejegg: Restore mistakenly-deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289981 (https://phabricator.wikimedia.org/T111677) [13:52:46] (03CR) 10Ejegg: [C:03+2] Restore mistakenly-deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289981 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [13:53:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1012 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:59:40] (03CR) 10CI reject: [V:04-1] Restore mistakenly-deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289981 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [13:59:42] (03PS2) 10Jgleeson: Add Native Payments config API endpoint [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289982 (https://phabricator.wikimedia.org/T426858) [14:00:01] wb ejegg [14:00:27] hope you're feelin better ! [14:00:32] school run time [14:01:41] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11941033 (10CHudson-WMF) >>! In T413905#11821142, @Ejegg wrote: > This would not affect any recurring contributions on the old paypal flow, that is, where contribution_recur.p... [14:01:51] (03CR) 10CI reject: [V:04-1] Add Native Payments config API endpoint [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289982 (https://phabricator.wikimedia.org/T426858) (owner: 10Jgleeson) [14:01:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb1007.frack.eqiad.wmnet:9004 for >30m - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb1007.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [14:03:37] (03Merged) 10jenkins-bot: Restore mistakenly-deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289981 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [14:06:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : frdb1007.frack.eqiad.wmnet:9004 for >30m - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb1007.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [14:06:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : frdb1007.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb1007.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [14:08:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1013 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [14:13:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1013 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [14:33:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1013 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [14:38:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1013 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:01:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1014 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:06:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1014 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:09:26] (03PS1) 10Ejegg: Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 [15:09:33] (03CR) 10Ejegg: [C:03+2] Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 (owner: 10Ejegg) [15:11:31] thanks jgleeson [15:14:42] (03CR) 10CI reject: [V:04-1] Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 (owner: 10Ejegg) [15:26:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1014 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:28:04] (03PS2) 10Ejegg: Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 (https://phabricator.wikimedia.org/T111677) [15:31:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1014 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:34:03] (03CR) 10Ejegg: [C:03+2] Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [15:39:59] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 14Fundraising-Backlog-Old, 10MediaWiki-extensions-DonationInterface, and 4 others: Some messages in the Donation extensions are outdated and should be removed - https://phabricator.wikimedia.org/T111677#11941459 (10Ejegg) OK, I've restored all the... [15:48:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1015 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:49:49] (03Merged) 10jenkins-bot: Restore translations of mistakenly deleted messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1290001 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [15:50:45] (03PS1) 10Ejegg: Restore mistakenly-deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290012 (https://phabricator.wikimedia.org/T111677) [15:50:48] (03PS1) 10Ejegg: Restore translations of mistakenly deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290013 (https://phabricator.wikimedia.org/T111677) [15:51:37] (03CR) 10Ejegg: [C:03+2] Restore mistakenly-deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290012 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [15:51:46] (03CR) 10Ejegg: [C:03+2] Restore translations of mistakenly deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290013 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [15:53:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1015 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [15:54:42] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07fr-current-sprint: Donor Portal documentation - https://phabricator.wikimedia.org/T410588#11941566 (10Damilare) a:03Damilare [16:02:09] (03PS4) 10Jgleeson: Add Native Payments config API endpoint [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1289982 (https://phabricator.wikimedia.org/T426858) [16:13:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1015 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [16:17:53] (03PS1) 10Jgleeson: dns fixes [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1290017 [16:18:05] RESOLVED: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1015 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [16:34:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [16:59:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 for >30m - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [17:09:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 for >30m - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [17:09:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : payments1012.frack.eqiad.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=payments1012.frack.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [18:08:46] 14Fundraising Sprint - GNU England Shaker dresser, 14Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Fraud: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11942100 (10Ejegg) Yes, it shoul... [18:40:10] 14Fundraising Sprint - GNU England Shaker dresser, 14Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Fraud: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11942209 (10MBeat33) Thanks, @Ej... [18:41:21] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 07fr-current-sprint: Delete enrollment for cancelled pix recurring - https://phabricator.wikimedia.org/T424245#11942210 (10AnnWF) a:03AnnWF [18:44:07] (03PS1) 10Ejegg: Fix undefined var in refund QC [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290041 [18:46:23] (03CR) 10Ejegg: [V:03+2 C:03+2] Restore mistakenly-deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290012 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [18:46:33] (03CR) 10Ejegg: [V:03+2 C:03+2] Restore translations of mistakenly deleted messages [extensions/DonationInterface] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1290013 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [18:55:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1016 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [19:00:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1016 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [19:00:32] 10Fundraising analytics stack, 10FR-Tech-Analytics: One-time import of SurveyMonkey data to Minio - https://phabricator.wikimedia.org/T424093#11942343 (10AStein-WMF) Cassie and I agreed on a normalized data structure and the data is now in Trino. Since this data will only be used until the surveymonkey api pip... [19:00:41] 10Fundraising analytics stack, 10FR-Tech-Analytics: One-time import of SurveyMonkey data to Minio - https://phabricator.wikimedia.org/T424093#11942344 (10AStein-WMF) 05Open→03Resolved [19:04:11] FIRING: NodeDown: Node frdb1004 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:05:17] ^^^ that's me. it'll be up in a sec. [19:09:11] RESOLVED: NodeDown: Node frdb1004 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:00:30] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 14Fundraising-Backlog-Old, 10MediaWiki-extensions-DonationInterface, and 3 others: Some messages in the Donation extensions are outdated and should be removed - https://phabricator.wikimedia.org/T111677#11942516 (10Ejegg) OK @Pcoombe , all of the... [21:05:43] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11942722 (10Ejegg) Oops, sorry, 19, 20 are the IDs in the local development environment - in production paypal IDs are 6, 7 [21:06:11] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11942725 (10Ejegg) @CHudson-WMF oops forgot to tag you on that last comment [21:27:00] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: fasw2-c8a-codfw:xe-0/0/47 low RX power - https://phabricator.wikimedia.org/T426824#11942813 (10Jhancock.wm) i can get this one in the morning if Jeff or Dallas is around and want to coordinate. [21:43:52] (03PS1) 10Eileen: Update civitoken [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290072 [21:44:48] (03PS1) 10Eileen: Update contact layout editor extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290073 [21:47:02] (03PS2) 10Eileen: Update civitoken [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290072 [21:47:02] (03PS2) 10Eileen: Update contact layout editor extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290073 [21:47:02] (03PS1) 10Eileen: Update recentmenu [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290074 [21:52:07] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11942891 (10Cstone) @CHudson-WMF a lot of these questions overlaps with what I was trying to work on that fr-tech dashboard, I think I was trying to graph what elliott wants t... [22:09:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Look into adyen recurrings that failed with FRAUD on May 12 - https://phabricator.wikimedia.org/T426916 (10Cstone) 03NEW [22:11:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:16:24] (03CR) 10Cstone: [C:03+2] Include matching_gift_organization in csv file [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289461 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [22:17:01] (03Merged) 10jenkins-bot: Include matching_gift_organization in csv file [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1289461 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [22:20:56] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1290077 [22:21:05] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1290077 (owner: 10Eileen) [22:21:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:21:32] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1290077 (owner: 10Eileen) [22:22:30] !log SmashPig upgraded from df441f78 to 252b6cfa [22:22:30] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [22:35:08] (03PS1) 10Eileen: Note another 'found' field [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1290079 [23:09:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [23:21:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [23:26:39] RESOLVED: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [23:30:10] ejegg|afk: I recall you mentioned something about unused/legacy event logging cod ein CN being ripe for removal. Is there a task for this and/or can you elaborate on what specifically? I think that might be a nice area to clean up next. [23:40:56] (03CR) 10Eileen: [C:03+2] Only normalize currency if it hasn't been normalized yet [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [23:44:47] 10fundraising-tech-ops: Deactivate fundraising accounts for jmoreira - https://phabricator.wikimedia.org/T426923 (10Dwisehaupt) 03NEW [23:47:36] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1290091 [23:47:55] (03PS1) 10Eileen: Move away from vague 'gross', 'fee', 'currency' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1290092 [23:49:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [23:53:57] 10fundraising-tech-ops: Deactivate fundraising accounts for jmoreira - https://phabricator.wikimedia.org/T426923#11943298 (10Dwisehaupt) [23:54:37] 10fundraising-tech-ops: Deactivate fundraising accounts for jmoreira - https://phabricator.wikimedia.org/T426923#11943301 (10Dwisehaupt) [23:55:31] 10fundraising-tech-ops: Deactivate fundraising accounts for jmoreira - https://phabricator.wikimedia.org/T426923#11943302 (10Dwisehaupt) [23:56:09] 10fundraising-tech-ops: Deactivate fundraising accounts for jmoreira - https://phabricator.wikimedia.org/T426923#11943304 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt ssl client certificate revoked. civi and metabase accounts set to disabled. [23:56:50] (03CR) 10Cstone: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1290091 (owner: 10Cstone) [23:57:45] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1290091 (owner: 10Cstone)