[01:23:05] FIRING: RAIDWarning: No RAID report for PERC/? on ) - https://alerts.wikimedia.org/?q=alertname%3DRAIDWarning [01:28:43] ^^^ that is an odd alert, i'm going to ack it for now as it isn't reporting the host and needs digging into. that can be for tomorrow. [02:45:35] (03CR) 10Cstone: [C:03+2] Delete obsolete utm_source fix [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279459 (owner: 10Ejegg) [02:47:33] (03Merged) 10jenkins-bot: Delete obsolete utm_source fix [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279459 (owner: 10Ejegg) [02:58:33] (03CR) 10Cstone: [C:03+2] Send landing_page as top-level c_t field [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279468 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [02:58:58] (03CR) 10Cstone: [C:03+2] Remove obsolete utm_source_id [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279480 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [03:00:30] (03Merged) 10jenkins-bot: Send landing_page as top-level c_t field [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279468 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [03:00:52] (03Merged) 10jenkins-bot: Remove obsolete utm_source_id [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1279480 (https://phabricator.wikimedia.org/T421028) (owner: 10Ejegg) [04:46:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:56:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:29:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Recurring-Donations: Some PayPal recurrings recorded with bad conversion rate - https://phabricator.wikimedia.org/T426098#11915665 (10Cstone) This was bothering me so I dug into it more but now I am more confused, this is where I got to tonight: This... [05:36:41] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1286588 (owner: 10L10n-bot) [12:28:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add a campaign to email activities for double opt in emails - https://phabricator.wikimedia.org/T422395#11917126 (10Lars) [12:29:00] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 07Epic, 13Patch-For-Review: Epic: Double opt-in - https://phabricator.wikimedia.org/T407017#11917127 (10Lars) [12:57:38] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change last donation amount, last donation original currency and last donation currency from foundation to all funds - https://phabricator.wikimedia.org/T426189 (10Lars) 03NEW [13:00:17] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change last donation amount, last donation original currency and last donation currency from foundation to all funds - https://phabricator.wikimedia.org/T426189#11917250 (10Lars) a:03Lars [13:01:12] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change last donation amount, last donation original currency and last donation currency from foundation to all funds - https://phabricator.wikimedia.org/T426189#11917255 (10Lars) [13:05:48] (03PS1) 10Lars SG: Change last donation amount, last donation original currency and last donation currency from foundation to all funds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286918 (https://phabricator.wikimedia.org/T426189) [13:08:05] If anyone could give this ^^^ a quick +2, it would be much appreciated. Just a simple change, but would be nice to deploy and then regenerate the triggers with this included. [13:38:54] (03CR) 10Krinkle: Move ext.centralNotice.startUp from onBeforePageDisplay to onSiteNoticeAfter (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1285504 (https://phabricator.wikimedia.org/T422082) (owner: 10Krinkle) [13:40:19] looking larssandergreen [13:42:40] (03CR) 10Ejegg: [C:03+2] Change last donation amount, last donation original currency and last donation currency from foundation to all funds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286918 (https://phabricator.wikimedia.org/T426189) (owner: 10Lars SG) [13:46:00] thanks ejegg [13:46:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:47:46] ^^^ I'm aware of the puppet thing, working on disabling nagios and there are some corner cases [14:01:10] RESOLVED: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:04:54] (03PS2) 10Ejegg: Use trixie for donut and core containers [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1282980 (https://phabricator.wikimedia.org/T424774) [14:04:56] (03CR) 10Ejegg: [V:03+2] Use trixie for donut and core containers [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1282980 (https://phabricator.wikimedia.org/T424774) (owner: 10Ejegg) [14:05:19] (03PS2) 10Ejegg: Update python containers to use trixie images [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1282987 (https://phabricator.wikimedia.org/T424774) [14:05:23] (03CR) 10Ejegg: [V:03+2 C:03+2] Update python containers to use trixie images [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1282987 (https://phabricator.wikimedia.org/T424774) (owner: 10Ejegg) [14:06:40] (03Merged) 10jenkins-bot: Change last donation amount, last donation original currency and last donation currency from foundation to all funds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286918 (https://phabricator.wikimedia.org/T426189) (owner: 10Lars SG) [14:11:09] (03PS1) 10Lars SG: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1286945 [14:12:50] (03PS2) 10Ejegg: Use trixie / PHP8.4 images for CiviCRM [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1283626 (https://phabricator.wikimedia.org/T424774) [14:12:52] (03CR) 10Ejegg: [V:03+2] Use trixie / PHP8.4 images for CiviCRM [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1283626 (https://phabricator.wikimedia.org/T424774) (owner: 10Ejegg) [14:13:52] (03CR) 10Lars SG: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1286945 (owner: 10Lars SG) [14:14:46] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1286945 (owner: 10Lars SG) [14:20:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:33:05] FIRING: RAIDWarning: No RAID report for PERC/? on ) - https://alerts.wikimedia.org/?q=alertname%3DRAIDWarning [14:43:09] (03PS4) 10Ejegg: Subclass Adyen response mapper for approvePayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) [14:43:09] (03PS4) 10Ejegg: Subclass Adyen response mapper for refundPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (https://phabricator.wikimedia.org/T423982) [14:43:11] (03PS2) 10Ejegg: Subclass Adyen response mapper for createPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284663 (https://phabricator.wikimedia.org/T423982) [14:43:14] (03PS1) 10Ejegg: Namespace Adyen response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286950 (https://phabricator.wikimedia.org/T423982) [14:43:32] ok, I redid that patch chain for ease of reviewing [14:43:42] I'll abandon the squashed patch [14:43:55] (03CR) 10CI reject: [V:04-1] Subclass Adyen response mapper for refundPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [14:43:58] (03CR) 10CI reject: [V:04-1] Subclass Adyen response mapper for approvePayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [14:44:09] (03CR) 10CI reject: [V:04-1] Subclass Adyen response mapper for createPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284663 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [14:44:14] gah [14:46:20] (03PS5) 10Ejegg: Subclass Adyen response mapper for approvePayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) [14:46:20] (03PS5) 10Ejegg: Subclass Adyen response mapper for refundPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (https://phabricator.wikimedia.org/T423982) [14:46:20] (03PS3) 10Ejegg: Subclass Adyen response mapper for createPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284663 (https://phabricator.wikimedia.org/T423982) [14:48:32] (03Abandoned) 10Ejegg: Namespace and subclass Adyen response mappers [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284664 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [15:13:00] (03PS1) 10Ejegg: Update psr/log to 3.0, remove Amazon Pay 1.0 SDK [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286960 (https://phabricator.wikimedia.org/T424497) [15:25:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [15:33:44] 10fundraising-tech-ops, 10observability: Deprecate Fundraising nsca icinga alert collection - https://phabricator.wikimedia.org/T425424#11918213 (10Jgreen) We've disabled all the passive checks, so there should be nothing incoming to the nsca server from FR anymore. [15:38:05] RESOLVED: RAIDWarning: No RAID report for PERC/? on ) - https://alerts.wikimedia.org/?q=alertname%3DRAIDWarning [15:57:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:02:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:06:22] (03PS7) 10Eileen: Use fileName (only) to track file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) [16:06:25] (03CR) 10Ejegg: [C:03+2] Use fileName (only) to track file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [16:07:08] (03CR) 10Jgleeson: [C:03+2] Namespace Adyen response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286950 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:07:44] (03Merged) 10jenkins-bot: Namespace Adyen response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286950 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:08:21] (03CR) 10Jgleeson: [C:03+2] Subclass Adyen response mapper for approvePayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:08:58] (03Merged) 10jenkins-bot: Subclass Adyen response mapper for approvePayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:08:59] (03CR) 10Jgleeson: [C:03+2] Subclass Adyen response mapper for refundPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:09:39] (03Merged) 10jenkins-bot: Subclass Adyen response mapper for refundPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:25:50] (03CR) 10Jgleeson: [C:03+2] "Restructure looks good to me! thanks" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284663 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:26:29] (03Merged) 10jenkins-bot: Subclass Adyen response mapper for createPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1284663 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [16:30:57] (03Merged) 10jenkins-bot: Use fileName (only) to track file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [17:01:55] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - https://phabricator.wikimedia.org/T426230 (10SHust) 03NEW [17:08:25] (03PS3) 10Ejegg: Update php-queue and predis dependencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283843 (https://phabricator.wikimedia.org/T425578) [17:08:25] (03PS2) 10Ejegg: Update psr/cache to 3.0 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286425 (https://phabricator.wikimedia.org/T425578) [17:08:25] (03PS2) 10Ejegg: Update psr/log to 3.0, remove Amazon Pay 1.0 SDK [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286960 (https://phabricator.wikimedia.org/T424497) [18:04:13] (03CR) 10Ejegg: "Seeing this failure message running locally "fgetcsv(): the $escape parameter must be provided as its default value will change". Guessing" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [18:13:33] (03PS1) 10Ejegg: Fix fgetcsv calls [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286994 (https://phabricator.wikimedia.org/T419044) [18:14:28] fr-tech ^^^ is pretty trivial but is needed for PHP 8.4 [18:14:57] all the other fgetcsv calls in smashpig already look like that [18:15:14] huh [18:16:49] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286994 (https://phabricator.wikimedia.org/T419044) (owner: 10Ejegg) [18:17:02] thanks jgleeson [18:17:09] NP [18:17:11] and thanks for the CR on all those mapper patches! [18:17:12] -caps [18:17:17] X2 [18:17:19] jeez [18:17:21] -caps [18:17:21] ha [18:17:28] (03Merged) 10jenkins-bot: Fix fgetcsv calls [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1286994 (https://phabricator.wikimedia.org/T419044) (owner: 10Ejegg) [18:18:07] I'll tag that to see if it gets the tests passing at the CRM level [18:23:53] /ac [18:24:30] (03PS1) 10Ejegg: Update smash-pig and other libs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286998 [18:26:53] (03CR) 10Ejegg: [C:03+2] Update smash-pig and other libs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286998 (owner: 10Ejegg) [18:27:42] (03PS2) 10Eileen: Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) [18:32:46] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal: Get our local civiproxy up to date with upstream - https://phabricator.wikimedia.org/T426078#11919025 (10Ejegg) Looks like main still doesn't have a way to switch off cert validation for development, so we WILL still need the overlay... [18:38:11] FIRING: NodeDown: Node payments1005 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1005 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:39:06] ^^^ that's me, it's rebooting. guess the silence expired. :) [18:43:05] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - https://phabricator.wikimedia.org/T426230#11919048 (10Lars) @SHust In this case, we automatically identified the donor as the same person as the existing contact when we processed their recent donation, so there... [18:43:11] RESOLVED: NodeDown: Node payments1005 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1005 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:43:26] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - https://phabricator.wikimedia.org/T426230#11919051 (10Lars) [18:50:28] (03Merged) 10jenkins-bot: Update smash-pig and other libs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1286998 (owner: 10Ejegg) [18:50:48] (03CR) 10CI reject: [V:04-1] Add Chariot audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281984 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [19:02:13] fr-tech, kid attack. relocating now, be a few mins late [19:04:58] (03PS1) 10Lars SG: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) [19:24:44] (03PS2) 10Lars SG: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287005 (https://phabricator.wikimedia.org/T426189) [19:54:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - https://phabricator.wikimedia.org/T426230#11919390 (10SHust) Thanks for the clarification, @Lars. I get the logic when a donor uses different payment providers (like PayPal vs... [20:04:19] 06Fundraising-Backlog, 06FR-donorrelations: Civi: indicate Forget Me request was performed on the contact summary - https://phabricator.wikimedia.org/T426254 (10MBeat33) 03NEW [20:04:29] 06Fundraising-Backlog, 06FR-donorrelations: Civi: indicate Forget Me request was performed on the contact summary - https://phabricator.wikimedia.org/T426254#11919439 (10MBeat33) p:05Triage→03Low [20:13:29] eileen: if you wanted to take a look at the trigger update, since you're back online: https://gerrit.wikimedia.org/r/1287005 [20:16:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make payments_fraud tables available in SK and add to Fraud Scan SK - https://phabricator.wikimedia.org/T426255 (10Lars) 03NEW [20:17:08] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make payments_fraud tables available in SK and add to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11919520 (10Lars) p:05Triage→03Medium [20:17:29] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make payments_fraud tables available in SK and add to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11919522 (10Lars) [20:28:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - due to contact matching in donation queue and emails being overwritten - https://phabricator.wikimedia.org/T426230#11919575 (10Lars) [20:30:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - due to contact matching in donation queue and emails being overwritten - https://phabricator.wikimedia.org/T426230#11919585 (10Lars) @Shust That does sound reasonable, we'll jus... [20:37:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - due to contact matching in donation queue and emails being overwritten - https://phabricator.wikimedia.org/T426230#11919664 (10SBurnett-WMF) I find that expansion to be an elega... [20:47:10] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on fransw2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:49:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Odd Civi behavior - due to contact matching in donation queue and emails being overwritten - https://phabricator.wikimedia.org/T426230#11919788 (10SHust) I'll be there, thank you! [20:52:10] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on fransw2001.frack.codfw.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:08:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [21:15:13] mariadb fixed up. [21:15:21] :) [21:15:56] the PuppetConstantChange is due to some notify stanzas. need to talk with jeff about their intended purpose before i just rip them out. have ack'd it through tomorrow morning. [21:18:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [23:58:11] FIRING: NodeDown: Node frpm1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frpm1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:58:46] ^^^ that's me [23:59:38] it's already back up.