[00:01:49] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11442558 (10Dwisehaupt) [00:15:06] (03Merged) 10jenkins-bot: Prioritise order retrieval by gateway_txn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216566 (https://phabricator.wikimedia.org/T405691) (owner: 10Eileen) [00:15:49] (03Merged) 10jenkins-bot: Readability fixes on batch status email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216568 (owner: 10Eileen) [00:18:47] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216684 [00:19:29] (03PS2) 10Eileen: Ensure that gravy venmo are 'found' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211935 (https://phabricator.wikimedia.org/T411143) [00:20:17] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216684 (owner: 10Eileen) [00:21:00] !log civicrm upgraded from 2dfecb38 to 86784b37 [00:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:37:07] (03CR) 10CI reject: [V:04-1] Ensure that gravy venmo are 'found' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211935 (https://phabricator.wikimedia.org/T411143) (owner: 10Eileen) [01:59:04] cstone: or ejegg could I get a +2 on this - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1216671 - I want to see if it clears with that one [01:59:16] (03Abandoned) 10Eileen: more comment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216678 (owner: 10Eileen) [02:00:25] sure eileen [02:00:32] (03CR) 10Cstone: [C:03+2] Continue mole-war [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216671 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [02:00:47] thanks cstone - I'm less sure about the follow on - I think ejegg might have thoughts [02:02:48] (03PS3) 10Eileen: Add test for fix to Ensure that gravy venmo are 'found' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211935 (https://phabricator.wikimedia.org/T411143) [02:06:45] hi eileen, let me take a look [02:08:26] ok, let's see, so getExistingContribution wouldn't pick up something with a different status than the current? [02:08:39] So we're not using that to get the parent for refunds, I guess? [02:09:01] sorry, takes me a little while to dive back into that code each time [02:10:37] eileen: in getExistingContributionID, won't isStatusChanged always be false now? [02:10:57] ohh - let me double check ... [02:10:59] or rather won't it always be false? [02:11:16] if getExistingCOntribution will only ever get a contribution with the current status? [02:11:52] oh sorry, that's only if ->isRefund or ->isChargeback [02:12:02] nvm eileen, let me keep reading here [02:16:25] (03Merged) 10jenkins-bot: Continue mole-war [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216671 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [02:17:23] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216696 [02:17:46] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216696 (owner: 10Eileen) [02:18:00] yep, looks good to me too cstone and eileen ! [02:18:35] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216696 (owner: 10Eileen) [02:19:17] !log civicrm upgraded from 86784b37 to f66aaff7 [02:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:19:44] ejegg: what about the follow on? I think something like that makes sense but I'm worried it doesn't work on a word count pov [02:24:16] oh hah, sure. I guess we'll know when we need to adjust the from_unique_id to handle those things beyond REFUND and RECURRING as the first $part [02:24:40] actually, I'll just do a further follow-on to add that in case [02:25:14] (03PS4) 10Eileen: Disambiguate trxn_id for extra refunds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216675 (https://phabricator.wikimedia.org/T411967) [02:25:20] (03CR) 10Ejegg: [C:03+2] Disambiguate trxn_id for extra refunds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216675 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [02:27:04] dang I don't think that last one did work right - it's found 17 missing in the first 10k - might be because I got stricter as I went along ... [02:27:10] & the data didn't [02:29:15] oh first one is a chargeback reversal... [02:30:04] (03PS1) 10Ejegg: Make WMFTransaction::from_unique_id work for new types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216700 [02:32:52] (03CR) 10Eileen: [C:03+2] "makes sense" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216700 (owner: 10Ejegg) [02:44:19] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Consider adding ability to convert Gravy IDs back & forth - https://phabricator.wikimedia.org/T411745#11442778 (10Eileenmcnaughton) Just noting that there are 21 payment processor reconciliation IDs which have been assigned to more than... [02:46:11] eileen: on that phab ^ theres some root bug thats letting a bunch of transactions happen all at once thats causing it in the first place right? [02:46:50] (03Merged) 10jenkins-bot: Disambiguate trxn_id for extra refunds [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216675 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [02:48:49] (03Merged) 10jenkins-bot: Make WMFTransaction::from_unique_id work for new types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216700 (owner: 10Ejegg) [03:08:41] cstone: at least one of them has the contribution once in USD and once in CAD [03:08:54] so from that little batch of wrong-currency recorded things I guess? [03:09:04] huhh [03:09:59] oh but the adyen txn id and reconciliation id are the same, though the gateway_txn_id is different??? [03:10:02] cid 25110915 [03:10:12] reconciliation ID 5icDq8fqFkPB7cvhzqD41x [03:10:38] is the duplicate task linked to that [03:11:01] it's in that list you just ^^^ed [03:11:21] * ejegg|away needs to go to bed, not down this rabbit hole [03:11:23] sorry... [03:11:42] haha good night ejegg|away [03:13:51] cstone: yeah - I'm hoping the root bug is fixed... [03:15:04] im trying to find that mistmatched id task i made a while ago [03:15:10] to see if this was the cause of that [03:21:58] (03PS1) 10Eileen: More debug output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216703 (https://phabricator.wikimedia.org/T411967) [03:22:49] cstone: another quick one ... https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1216703 (I could self-merge this one on the debug mandate but since you are about...) [03:24:40] OH EILEEN i forgot [03:24:44] you got $51 :) [03:25:07] cstone: OMG - is that the class action??? [03:25:10] $51.84 to be specific [03:25:32] woohoo - we will have to put it towards a team ... erm ... icecream somewhere someplace [03:25:46] yep!!! courtesy of of Gregory V Tubi Inc [03:25:50] (not quite enough for a team dinner ) [03:25:55] 2 apps [03:26:10] (03CR) 10Cstone: [C:03+2] More debug output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216703 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [03:26:23] at dinner hah [03:26:27] team giant bowl of icecream [03:26:46] cstone: probably the hourly rate on figuring out how to get it was around $10 an hour... [03:26:52] haha [03:27:08] complicated enough that you got $51 instead of $2 [03:27:16] lol [03:31:23] eileen: this is the civi part of the sms opt in from the payments form [03:31:23] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1214236 [03:31:33] in the WIP i just copied over what it was doing in the acoustic code [03:31:54] does it make sense to you to do it that way? [03:32:24] the on will be going way just there for the wip [03:37:16] (03PS1) 10Eileen: Only check for status change on negative [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216704 (https://phabricator.wikimedia.org/T411967) [03:38:27] (03PS2) 10Eileen: Only check for status change on negative [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216704 (https://phabricator.wikimedia.org/T411967) [03:38:29] cstone: looking ... also I think ^^ was what I got wrong last time... [03:42:50] (03Merged) 10jenkins-bot: More debug output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216703 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [03:44:56] (03CR) 10Cstone: [C:03+2] Only check for status change on negative [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216704 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [03:45:50] (03CR) 10Eileen: "It seems generally OK but would only work for the Update not the Create. I'm wondering if it makes sense to create the activity in a hook " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1214236 (https://phabricator.wikimedia.org/T380106) (owner: 10Cstone) [03:46:28] hmm it was making it on a new contact locally eileen ^ [03:46:39] cstone: oh ok? [03:47:44] (03PS12) 10Cstone: Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) [03:48:01] if it is then maybe there is a hook cos the update code doesn't look like it would create the consent record [03:48:11] hmm i need to refactor its name anyway ill step through it again [03:49:44] I see in Omniphone.update we use `sms_consent_status` as the field name - might make sense to aligh 'opted_in' => $details['sms_consent_status'] === 'OPTED-IN', [03:50:06] (03CR) 10CI reject: [V:04-1] Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) (owner: 10Cstone) [03:50:26] although the 'details' there are from Acoustic [03:52:11] we could share the code in Omniphone_Update from `$details` on if we tried [03:55:45] (03PS13) 10Cstone: Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) [03:57:08] would we want to treat these consents differently in any way besides their source eileen ? [04:01:34] well for Acoustic we do a look up & get additional data - obviously not that. But apart from that I think the source is enough..... [04:01:45] (03Merged) 10jenkins-bot: Only check for status change on negative [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216704 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [04:02:34] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216706 [04:05:00] (03PS1) 10Lars SG: Add BulkReOptIn API action for Acoustic contacts. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) [04:05:39] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216706 (owner: 10Eileen) [04:06:19] !log civicrm upgraded from f66aaff7 to e0867392 [04:06:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:06:30] (03CR) 10Lars SG: Add BulkReOptIn API action for Acoustic contacts. (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [04:07:46] eileen: this one could use review, so I can get the process going to re opt in emails ^^^ [04:08:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Should we opt back in contacts if they unsubscribe and then later RML? - https://phabricator.wikimedia.org/T411906#11442824 (10MSuijkerbuijk_WMF) Hi @Lars Yes we should opt them back in. This applies to both cases - bec... [04:16:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Should we opt back in contacts if they unsubscribe and then later RML? - https://phabricator.wikimedia.org/T411906#11442831 (10Lars) @MSuijkerbuijk_WMF For point 2, most of them will have donated in the past (as these ar... [04:18:09] (03PS1) 10Eileen: Check for chargeback reversals first & if-else-it [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216709 (https://phabricator.wikimedia.org/T411967) [04:18:46] ok larssandergreen I'll get a cuppa & take a look [04:18:59] thanks! [04:41:21] (03CR) 10Cstone: [C:03+2] Check for chargeback reversals first & if-else-it [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216709 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [04:56:14] (03CR) 10Eileen: "Some superficial comments - I'm going to have a go at running it next" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [04:57:35] (03Merged) 10jenkins-bot: Check for chargeback reversals first & if-else-it [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216709 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [05:02:55] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216711 [05:03:11] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216711 (owner: 10Eileen) [05:04:01] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1216711 (owner: 10Eileen) [05:04:52] !log civicrm upgraded from e0867392 to bdf84821 [05:04:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:15:24] well I'm opted in - that worked [05:38:38] (03CR) 10Eileen: "I successfully opted myself out so I'll look at the bulk action tomorrow" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [05:44:17] (03PS2) 10Lars SG: Add BulkReOptIn API action for Acoustic contacts. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) [05:45:13] (03CR) 10Lars SG: Add BulkReOptIn API action for Acoustic contacts. (032 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [05:46:03] (03PS3) 10Lars SG: Add BulkReOptIn API action for Acoustic contacts. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) [05:46:24] eileen: re the is_opt_out and opt_in, I can combine them if you think it makes more sense. I guess in my mind I'm mapping is_opt_out to the Civi fields and trying not to confuse things by also jamming the opt in on top of that. But if you think it makes more sense to combine, that's fine. [05:46:44] larssandergreen: I think my head hurts :-) [05:48:16] I'll ponder it - we have tech talk tomorrow early-ish [05:53:50] eileen: sounds good. [05:54:23] (03CR) 10Lars SG: "Added handling for no limit as well, which I had forgotten." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [05:55:53] (03PS14) 10Cstone: Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) [05:56:18] larssandergreen: we are online too late :P [05:57:25] (03CR) 10Lars SG: "Part of my thought in separating is_opt_in from is_opt_out is also making sure we are very explicit about what we are trying to do, becaus" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216707 (https://phabricator.wikimedia.org/T409958) (owner: 10Lars SG) [05:57:51] (03CR) 10Cstone: "Name changed, I thought I had named it to match the others then realized it wasn't when I was writing the civi code" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) (owner: 10Cstone) [05:57:55] (03CR) 10CI reject: [V:04-1] Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) (owner: 10Cstone) [05:57:59] cstone: well, i'm signing off anyways [05:58:17] me too have a good night larssandergreen [05:58:25] and rest of your day eileen [05:59:39] (03PS15) 10Cstone: Add variant for SMS optin [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1193298 (https://phabricator.wikimedia.org/T380106) [07:46:17] (03PS1) 10Eileen: Do not allow other transactions to pick up ChargebackReversals [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216725 (https://phabricator.wikimedia.org/T411967) [09:07:17] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Remove inaccessible inputs from Payments wiki when donor is redirected automatically - https://phabricator.wikimedia.org/T411902#11443005 (10Pcoombe) →14Duplicate dup:03T410318 [09:07:18] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 06FR-donorrelations, 0725 Dec Q3 Planning: PayPal flow doesn't allow employer field to be utilized - https://phabricator.wikimedia.org/T410318#11443007 (10Pcoombe) [09:17:47] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Clarify and/or rethink centralnotice device settings - https://phabricator.wikimedia.org/T411908#11443033 (10Pcoombe) It's not a question of "working around", it's just inaccurate and confusing for users. There's no real urgency from a fundraising p... [11:57:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Should we opt back in contacts if they unsubscribe and then later RML? - https://phabricator.wikimedia.org/T411906#11443658 (10MSuijkerbuijk_WMF) @Lars That makes sense. To me, this works as usual and we can opt them bac... [12:00:09] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 06FR-donorrelations, 0725 Dec Q3 Planning: Feature request assessment: Zendesk macro updater - https://phabricator.wikimedia.org/T387503#11443661 (10krobinson) I think mid-late March feels realistic - generally we won't have the translations done before th... [12:03:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11443691 (10krobinson) Thanks @Lars - can you confirm what the source of the missing data is? I wasn't clear fr... [12:39:09] 10Fundraising - Civi Standalone Issues: Japan Recurring Gifts September - https://phabricator.wikimedia.org/T412025#11443762 (10Aklapper) Hi @ERoden-WMF, please associate one or more active project tags with this task (via the {nav name=Add Action... > Change Project Tags} dropdown), so the task can be found whe... [12:54:30] 06Fundraising-Backlog: "View in Browser" feature for Civi Mailings - https://phabricator.wikimedia.org/T412111 (10ppenloglou) 03NEW [13:19:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Last donation currency not updated when processing recurring donations - https://phabricator.wikimedia.org/T405690#11443887 (10Damilare) a:03Damilare [13:41:43] (03CR) 10Damilare Adedoyin: [C:03+2] Do not allow other transactions to pick up ChargebackReversals [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216725 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [13:42:47] (03CR) 10Damilare Adedoyin: [C:03+2] Add test for fix to Ensure that gravy venmo are 'found' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211935 (https://phabricator.wikimedia.org/T411143) (owner: 10Eileen) [14:01:36] 06Fundraising-Backlog: Japan Recurring Gifts September - https://phabricator.wikimedia.org/T412025#11443990 (10ERoden-WMF) [14:04:14] (03Merged) 10jenkins-bot: Do not allow other transactions to pick up ChargebackReversals [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216725 (https://phabricator.wikimedia.org/T411967) (owner: 10Eileen) [14:04:17] (03Merged) 10jenkins-bot: Add test for fix to Ensure that gravy venmo are 'found' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211935 (https://phabricator.wikimedia.org/T411143) (owner: 10Eileen) [14:05:52] 14Fundraising Sprint: Vixen, 03Fundraising Sprint: Xiaolongbao, 06Fundraising-Backlog, 07fr-current-sprint: Add monitoring for number of transactions imported with |dup-XXXXX appended to the invoice ID - https://phabricator.wikimedia.org/T402316#11444001 (10Jgreen) 05Open→03Resolved a:03Jgreen >>... [14:22:32] (03PS3) 10Ejegg: Add donor langauge to prefs links [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204959 [15:19:50] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 06FR-donorrelations, 0725 Dec Q3 Planning: Feature request assessment: Zendesk macro updater - https://phabricator.wikimedia.org/T387503#11444189 (10SBorriello) Yes, late March sounds better to me, too. Thank you for your help! [15:24:35] 10fundraising-tech-ops: Reduce fundraising prometheus trino metric collection - https://phabricator.wikimedia.org/T412121 (10Jgreen) 03NEW [15:48:21] 06Fundraising-Backlog, 06FR-donorrelations: Error message formatting on mobile - https://phabricator.wikimedia.org/T412124 (10AMJohnson) 03NEW [15:49:24] 06Fundraising-Backlog, 06FR-donorrelations: Error message formatting on mobile - https://phabricator.wikimedia.org/T412124#11444279 (10AMJohnson) [16:00:33] 03Fundraising Sprint WWaally hoo, 14Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 4 others: Implement recurring donation monthly-to-annual conversion - https://phabricator.wikimedia.org/T388753#11444348 (10AKanji-WMF) Noting I tested this right now as part of QA... [16:00:46] 03Fundraising Sprint WWaally hoo, 14Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 4 others: Implement recurring donation monthly-to-annual conversion - https://phabricator.wikimedia.org/T388753#11444349 (10AKanji-WMF) 05Resolved→03Open [16:27:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fix civicrm/report/wmffraud/fredge results - https://phabricator.wikimedia.org/T411127#11444438 (10jgleeson) a:03jgleeson [16:35:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11444462 (10Lars) @krobinson I don't have a full understanding right now of how all the different ways someone... [16:42:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11444497 (10krobinson) Thanks for that extra context @Lars . So long as they are opted out in Acoustic and no... [17:06:11] FIRING: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [17:10:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:10:17] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:11:11] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:11:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:12:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:12:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:12:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:12:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 503 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 32 keys, up 25 days 15 hours - replication_delay is 503, memory use is 271.28M (peak 627.14M, 3.53% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:12:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 507 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 25 days 15 hours - replication_delay is 507, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.03%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:12:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 505 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 32 keys, up 25 days 15 hours - replication_delay is 505, memory use is 271.28M (peak 627.14M, 3.53% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:12:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 509 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 25 days 15 hours - replication_delay is 509, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:12:46] hmmm... wonder if there is a cross colo issue. [17:13:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:15:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:15:17] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:15:25] yeah. known issue on the network side. i'm goign to ack these. [17:16:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:16:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:17:06] ACKNOWLEDGEMENT - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:17:06] ACKNOWLEDGEMENT - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:17:06] ACKNOWLEDGEMENT - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:17:06] ACKNOWLEDGEMENT - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:17:06] ACKNOWLEDGEMENT - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:17:07] ACKNOWLEDGEMENT - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 503 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 32 keys, up 25 days 15 hours - replication_delay is 503, memory use is 271.28M (peak 627.14M, 3.53% of max, fragmentation 1.07%) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga. [17:17:07] .org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:17:07] ACKNOWLEDGEMENT - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 507 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 25 days 15 hours - replication_delay is 507, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.03%) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedi [17:17:07] -bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:17:08] ACKNOWLEDGEMENT - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 505 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 32 keys, up 25 days 15 hours - replication_delay is 505, memory use is 271.28M (peak 627.14M, 3.53% of max, fragmentation 1.07%) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga. [17:17:08] .org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:17:09] ACKNOWLEDGEMENT - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 509 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 25 days 15 hours - replication_delay is 509, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.07%) Dwisehaupt known - PfwCoreBGPDown network infrastructure-foundations (gnmi DEFAULT (null) VPN EXTERNAL ops BGP DEFAULT critical prometheus) https://icinga.wikimedi [17:25:13] RECOVERY - check_mysql on payments2006 is OK: Uptime: 2213569 Threads: 6 Questions: 9956518 Slow queries: 0 Opens: 148 Open tables: 142 Queries per second avg: 4.497 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:25:17] RECOVERY - check_mysql on payments2005 is OK: Uptime: 2213583 Threads: 6 Questions: 9915618 Slow queries: 0 Opens: 148 Open tables: 142 Queries per second avg: 4.479 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:26:09] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 2241712 Threads: 4 Questions: 418951950 Slow queries: 1273 Opens: 322860 Open tables: 1570 Queries per second avg: 186.889 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6728 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:26:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 2241873 Threads: 4 Questions: 417798494 Slow queries: 1277 Opens: 322443 Open tables: 1494 Queries per second avg: 186.361 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9676 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:27:11] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 1614105 Threads: 103 Questions: 607330029 Slow queries: 8772 Opens: 324848 Open tables: 2783 Queries per second avg: 376.264 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 285 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:27:15] RECOVERY - check_mysql on payments2004 is OK: Uptime: 2215048 Threads: 6 Questions: 9926001 Slow queries: 0 Opens: 148 Open tables: 142 Queries per second avg: 4.481 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:27:17] RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 28 keys, up 25 days 15 hours - replication_delay is 0, memory use is 268.12M (peak 627.14M, 3.50% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:27:19] RECOVERY - check_redis_donor_prefs on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 1 databases (db0) with 1 keys, up 25 days 15 hours - replication_delay is 1, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.01%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:27:19] RECOVERY - check_redis on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 29 keys, up 25 days 15 hours - replication_delay is 0, memory use is 268.18M (peak 627.14M, 3.50% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:27:21] RECOVERY - check_redis_donor_prefs on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 1 databases (db0) with 1 keys, up 25 days 15 hours - replication_delay is 3, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:28:11] RECOVERY - check_mysql on frdb1007 is OK: Uptime: 1709271 Threads: 4 Questions: 3364991876 Slow queries: 368 Opens: 324082 Open tables: 2227 Queries per second avg: 1968.670 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 397 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:31:26] 06Fundraising-Backlog, 10FR-Donor-portal: Determine work for test automation - Donor Portal - https://phabricator.wikimedia.org/T409209#11444764 (10AKanji-WMF) @KHill-WMF would you be able to liase with Sean on a sprint in Q3 where we could align on some test automation? [17:32:11] RECOVERY - check_mysql on frdata2002 is OK: Uptime: 2234397 Threads: 3 Questions: 1023115 Slow queries: 0 Opens: 135 Open tables: 129 Queries per second avg: 0.457 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:37:47] RESOLVED: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [17:42:53] 06Fundraising-Backlog: "View in Browser" feature for Civi Mailings - https://phabricator.wikimedia.org/T412111#11444798 (10Lars) @ppenloglou Unfortunately, we can't use this token as things stand now because we block all traffic to Civi that isn't logged in (and this is a CiviCRM URL). This is the same reason we... [17:43:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Can't delete mailing in CiviCRM - https://phabricator.wikimedia.org/T412144 (10Lars) 03NEW [17:50:43] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Chariot import mapping template - 2 soft credits - https://phabricator.wikimedia.org/T411555#11444863 (10MDemosWMF) @Lars Was wondering if in the meantime we would be able to create two import templates for linked payers we are... [17:56:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2584 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:01:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1244 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:06:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 2244273 Threads: 4 Questions: 423698233 Slow queries: 1282 Opens: 323441 Open tables: 1503 Queries per second avg: 188.790 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 156 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:53:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Last donation currency not updated when processing recurring donations - https://phabricator.wikimedia.org/T405690#11445073 (10Damilare) Having looked at this, I can see the last discrepancy for contributions from the Re... [20:24:05] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 0725 Dec Q3 Planning: Clarify and/or rethink centralnotice device settings - https://phabricator.wikimedia.org/T411908#11445253 (10AKanji-WMF) [20:29:01] (03PS1) 10Dwisehaupt: Shift to using bz2 compression from gzip [wikimedia/fundraising/tools/DjangoBannerStats] - 10https://gerrit.wikimedia.org/r/1216849 [20:38:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Can't delete mailing in CiviCRM - https://phabricator.wikimedia.org/T412144#11445524 (10Lars) p:05Triage→03Low [20:48:50] (03PS1) 10Eileen: Port upstream https://github.com/civicrm/civicrm-core/pull/34208 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216853 [20:51:26] ^^ is a super minor backport - should be an easy +2 [21:42:15] 03Fundraising Sprint: Xiaolongbao, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Missing Braintree transactions - https://phabricator.wikimedia.org/T411143#11445697 (10Eileenmcnaughton) I used isMakeMissing to create the second of the 2 missing above contri... [21:57:09] (03CR) 10Cstone: [C:03+2] Port upstream https://github.com/civicrm/civicrm-core/pull/34208 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216853 (owner: 10Eileen) [22:15:05] (03Merged) 10jenkins-bot: Port upstream https://github.com/civicrm/civicrm-core/pull/34208 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1216853 (owner: 10Eileen) [22:47:10] PROBLEM - check_load on fran1002 is CRITICAL: LOAD CRITICAL - total load average: 31.89, 24.74, 15.95 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1002&service=check_load [22:49:19] ACKNOWLEDGEMENT - check_load on fran1002 is CRITICAL: LOAD CRITICAL - total load average: 31.89, 24.74, 15.95 Dwisehaupt known - banner log processing tests. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1002&service=check_load [23:07:08] RECOVERY - check_load on fran1002 is OK: LOAD OK - total load average: 0.02, 0.53, 4.64 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1002&service=check_load