[00:40:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 9.69, 41.77, 27.04 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:45:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 17.29, 20.32 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:50:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 1.49, 8.29, 15.65 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:50:46] ^^ i'll put in an ack. the host is ok, it's just alerting on the 15 min load from an intensive job. [00:51:16] ACKNOWLEDGEMENT - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 1.49, 8.29, 15.65 Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:56:41] 10fundraising-tech-ops: Schedule fr-tech reboots for microcode updates - https://phabricator.wikimedia.org/T408742#11357036 (10Dwisehaupt) [00:56:57] 10fundraising-tech-ops: Schedule fr-tech reboots for microcode updates - https://phabricator.wikimedia.org/T408742#11357037 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt All reboots complete. [01:00:07] RECOVERY - check_load on franio2003 is OK: LOAD OK - total load average: 2.31, 2.56, 8.83 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [08:15:30] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203325 (owner: 10L10n-bot) [08:47:57] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203325 (owner: 10L10n-bot) [09:46:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11357620 (10krobinson) I think I have another one: https://wikimedia.zendesk.com/agent/tickets/1768899 / CID... [09:50:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11357687 (10krobinson) Ditto https://wikimedia.zendesk.com/agent/tickets/1769237 / CID 2289661, which was defin... [14:49:41] (03PS5) 10Wfan: Handle apple pay session by clear it after complete and cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199045 (https://phabricator.wikimedia.org/T407584) [14:49:43] (03CR) 10Ejegg: [C:03+2] Handle apple pay session by clear it after complete and cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199045 (https://phabricator.wikimedia.org/T407584) (owner: 10Wfan) [14:50:29] (03CR) 10Ejegg: [V:03+2 C:03+2] Update WP25 appeal for new image [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1202731 (owner: 10Ejegg) [14:55:50] (03Merged) 10jenkins-bot: Handle apple pay session by clear it after complete and cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199045 (https://phabricator.wikimedia.org/T407584) (owner: 10Wfan) [15:42:43] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, 07fr-acoustic: Suppression list logic - sanity check - https://phabricator.wikimedia.org/T325125#11359248 (10SBurnett-WMF) @AKanji-WMF - should I review this with Ilse & Mariana before we do a ton rewiring the donor portal? [15:55:44] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10fundraising-tech-ops, 07fr-current-sprint: FR-Tech FY2526Q2 maintenance window - https://phabricator.wikimedia.org/T364184#11359323 (10AStein-WMF) [16:08:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:09:57] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10FR-Donor-portal, 13Patch-For-Review: Donor Portal - Problems Logging In Flow - https://phabricator.wikimedia.org/T407453#11359363 (10KHill-WMF) >>! In T407453#11355588, @AnnWF wrote: > {F70012136} is mail to use link as below looks ok or you want me to... [16:13:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:13:26] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11359385 (10Lars) Thanks @krobinson. It looks like we aren't getting this data from Acoustic somehow, I will di... [16:18:17] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:38:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11359546 (10Lars) a:03Lars @Elbar53 can you share a link to the file and to the import in Civi? If you're crea... [16:38:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11359554 (10Lars) p:05Triage→03Medium [16:51:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11359651 (10Aklapper) [17:41:11] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal, and 2 others: Implement recurring donation monthly-to-annual conversion - https://phabricator.wikimedia.org/T388753#11359935 (10Ejegg) We could, but it would require us to instantiate a new gateway objec... [17:43:31] (03PS1) 10Ejegg: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1203497 [17:50:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11360013 (10Lars) @Elbar53 I've fixed this particular import so you can now see the errors. But this does seem t... [17:54:45] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1203497 (owner: 10Ejegg) [17:58:39] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1203497 (owner: 10Ejegg) [18:01:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11360045 (10SHust) @Lars, the unsubscribe info is showing under Change-Log and Activities for the CIDs, but it’... [18:17:26] larssandergreen: I'm seeing plenty of batch_merge queue items claimed in the civi logs - can we move T408748 to 'done'? [18:17:27] T408748: Coworker dedupe isn't running - https://phabricator.wikimedia.org/T408748 [18:27:09] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 485505MiB (10% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23689MiB (89% inode=99%): /var/tmp 408MiB [18:27:09] e=99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:29:07] 06Fundraising-Backlog, 07payments-orchestration: Add support for a new parameter to show a Pending Thank You (TY) page - https://phabricator.wikimedia.org/T409500#11360206 (10KHill-WMF) I've created a [[ https://docs.google.com/document/d/1LMy58ZuHHMjf4LBB8JyqShzms2AoyiTKwxlWkcbG6LM/edit?usp=sharing | draft of... [18:32:11] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 455030MiB (9% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23689MiB (89% inode=99%): /var/tmp 408MiB ( [18:32:11] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:37:09] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 424148MiB (9% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [18:37:09] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:38:33] 06Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Metabase Access Request - Olga Vasileva - https://phabricator.wikimedia.org/T404627#11360268 (10AStein-WMF) Olga can access metabase! [18:38:53] 06Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Metabase Access Request - Olga Vasileva - https://phabricator.wikimedia.org/T404627#11360273 (10AStein-WMF) 05Open→03Resolved [18:42:08] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 397654MiB (8% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [18:42:08] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:47:08] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 371183MiB (8% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [18:47:08] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:52:09] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 344577MiB (7% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [18:52:09] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:54:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11360328 (10Elbar53) Thanks Lars. I'll import this file today unless you want me to wait. [18:56:57] ejegg: That's good news, but it hasn't resolved the example Sandra provided in the Phab. Maybe still running through the backlog? [18:57:09] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 317273MiB (6% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [18:57:09] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [18:59:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Benevity import not showing errors - https://phabricator.wikimedia.org/T409744#11360356 (10Lars) @Elbar53 Definitely fine to go ahead, the issue is just with the display, no issues with the im... [19:02:07] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 289752MiB (6% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [19:02:07] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [19:07:07] PROBLEM - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 263197MiB (5% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB ( [19:07:07] =99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [19:07:54] ACKNOWLEDGEMENT - check_disk on frdb2003 is CRITICAL: DISK CRITICAL - free space: /srv 263197MiB (5% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp [19:07:54] 9% inode=99%): /var/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): Jeff_Green yes. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [19:11:30] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, 07fr-acoustic: Suppression list logic - sanity check - https://phabricator.wikimedia.org/T325125#11360407 (10AKanji-WMF) @SBurnett-WMF - adding in @Lars - this is about the master suppression list, right? and just a considerati... [19:16:41] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, 07fr-acoustic: Suppression list logic - sanity check - https://phabricator.wikimedia.org/T325125#11360431 (10Lars) @SBurnett-WMF and @AKanji-WMF this is already an active discussion in the context of how contacts come off of th... [19:27:07] RECOVERY - check_disk on frdb2003 is OK: DISK OK - free space: /srv 838302MiB (18% inode=99%): /dev 257270MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4438MiB (63% inode=89%): /dev/shm 257297MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 561MiB (82% inode=99%): /home 8177MiB (66% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25481MiB (96% inode=99%): /var/log 23688MiB (89% inode=99%): /var/tmp 408MiB (99% inode= [19:27:07] r/log/audit 408MiB (99% inode=99%): /run/user/573 51459MiB (100% inode=99%): /run/user/16798 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_disk [19:33:55] (03PS1) 10Lars SG: Add updateAnnualDonors API Action This ended up being too complex to handle with an API call directly (let alone from cv), so I've added an API action to handle it. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203508 (https://phabricator.wikimedia.org/T377438) [19:35:02] eileen: can you take a look at the above? [19:36:55] 10fundraising-tech-ops: Expired GPG key for mysql-tools repo - https://phabricator.wikimedia.org/T409609#11360496 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt Pulled the key from the ubuntu keyserver and added it to the config. Repo updates are happy again. [19:39:06] (03PS2) 10Ejegg: Fix typos in upstream matching gifts info [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202851 (https://phabricator.wikimedia.org/T404396) [19:47:33] oops, more deadlocks [19:48:05] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10FR-Donor-portal, 13Patch-For-Review: Donor Portal - Problems Logging In Flow - https://phabricator.wikimedia.org/T407453#11360551 (10KHill-WMF) Mailto email copy: [Subject line:] Problem accessing Wikimedia Foundation donor account [Email body copy:]... [19:50:38] larssandergreen: there were a couple of bad queries causing deadlocks - looks like activity searches. Do you want the query text pasted someplace? I've just killed them but saved the text. [19:51:02] oh, I'll just reply to the donations queue failmail with their text [20:13:18] ejegg: thanks, I've followed up with Desiree. [20:22:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11360626 (10Lars) a:03Lars [20:35:41] 03Fundraising Sprint: UTM_key lime pie, 06Fundraising-Backlog, 10FR-payments-general, 05MW-1.46-notes (1.46.0-wmf.2; 2025-11-12): PaymentsWiki W25 design test - https://phabricator.wikimedia.org/T407241#11360698 (10ppenloglou) Hey @Ejegg! I tried replicating your issue but with no luck unfortunately. H... [20:39:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Coworker dedupe isn't running - https://phabricator.wikimedia.org/T408748#11360712 (10Lars) p:05Triage→03Medium [20:39:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07payments-orchestration: Review Adyen/Gravy webhook recipients list. - https://phabricator.wikimedia.org/T408590#11360713 (10XenoRyet) p:05Triage→03Low [20:39:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11360715 (10Lars) p:05Triage→03Medium [20:40:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: refundfailed status from adyen - https://phabricator.wikimedia.org/T408443#11360730 (10XenoRyet) p:05Triage→03Medium [20:42:00] 06Fundraising-Backlog: Prevent [ change email ] from breaking onto multiple lines in email preferences centre - https://phabricator.wikimedia.org/T409576#11360740 (10Lars) a:03Lars [20:42:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Venmo issues from first round of testing - https://phabricator.wikimedia.org/T409599#11360742 (10AKanji-WMF) p:05Low→03High [20:43:28] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog: Prevent [ change email ] from breaking onto multiple lines in email preferences centre - https://phabricator.wikimedia.org/T409576#11360747 (10AKanji-WMF) [20:46:53] 06Fundraising-Backlog: Add (check?) currency minimums on gravy - https://phabricator.wikimedia.org/T409522#11360763 (10AKanji-WMF) @Pcoombe some of us think that you might have a list of minimums we can work off? If not - @RKumar_WMF @EMartin please let us know what the min would be on this - potentially we nee... [20:50:07] 06Fundraising-Backlog: Autorescue result tracking - https://phabricator.wikimedia.org/T409384#11360771 (10AKanji-WMF) Moved to next in triage @Cstone - let us know if there's a milestone specific to this [20:51:58] (03CR) 10Wfan: [C:03+2] Update exchange rates [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1201833 (owner: 10Ejegg) [20:52:33] (03Merged) 10jenkins-bot: Update exchange rates [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1201833 (owner: 10Ejegg) [20:53:42] 06Fundraising-Backlog: Add (check?) currency minimums on gravy - https://phabricator.wikimedia.org/T409522#11360783 (10AKanji-WMF) never mind @Pcoombe - looks like you pull this from us :) @RKumar_WMF @EMartin please confirm a minimum amount [20:55:05] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Get Paypal audit processing Intactable - https://phabricator.wikimedia.org/T409362#11360791 (10AKanji-WMF) Flagging anyone can work on this [20:55:46] 06Fundraising-Backlog, 07payments-orchestration: Add support for a new parameter to show a Pending Thank You (TY) page - https://phabricator.wikimedia.org/T409500#11360800 (10AnnWF) >>! In T409500#11360206, @KHill-WMF wrote: > I've created a [[ https://docs.google.com/document/d/1LMy58ZuHHMjf4LBB8JyqShzms2Aoyi... [21:02:33] 06Fundraising-Backlog: Add a cron job to update the local version currencyRate - https://phabricator.wikimedia.org/T409235#11360837 (10AKanji-WMF) - commit, merge file, tag smashpig version [21:04:28] 06Fundraising-Backlog: Add a cron job to update the local version currencyRate - https://phabricator.wikimedia.org/T409235#11360842 (10AnnWF) [21:05:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237#11360844 (10AKanji-WMF) p:05Triage→03Medium [21:11:34] 06Fundraising-Backlog, 07fr-payments-ACH: Recurring ACH looks like a success but then is actually failed - https://phabricator.wikimedia.org/T409471#11360859 (10AKanji-WMF) [21:11:37] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Add support for a new parameter to show a Pending Thank You (TY) page - https://phabricator.wikimedia.org/T409500#11360858 (10AKanji-WMF) [21:11:39] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Add support for a new parameter to show a Pending Thank You (TY) page - https://phabricator.wikimedia.org/T409500#11360860 (10AnnWF) [21:11:40] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Create Civi Contribution with Pending Status for ACH - https://phabricator.wikimedia.org/T409502#11360862 (10AKanji-WMF) [21:11:41] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Update Civi Record on status Pending from Webhook - https://phabricator.wikimedia.org/T409503#11360861 (10AKanji-WMF) [21:12:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-payments-ACH: Recurring ACH looks like a success but then is actually failed - https://phabricator.wikimedia.org/T409471#11360865 (10AnnWF) [21:13:20] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Update Civi Record on status Pending from Webhook - https://phabricator.wikimedia.org/T409503#11360869 (10AnnWF) [21:16:18] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Create Civi Contribution with Pending Status for ACH - https://phabricator.wikimedia.org/T409502#11360874 (10AnnWF) [21:26:12] (03PS1) 10Ejegg: Show 'unsupported card' message for AmEx in India [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203551 (https://phabricator.wikimedia.org/T409211) [21:29:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review: Validation for AmEx BINS in India - https://phabricator.wikimedia.org/T409211#11360924 (10Ejegg) [21:41:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Unsubscribe Description Not Updating - https://phabricator.wikimedia.org/T409478#11360990 (10Lars) @krobinson and @SHust There are actually two types of unsubscribes that we get from Acoustic... [21:46:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237#11361013 (10Ejegg) At least a few of these are manually entered b9e97347-db8a-4214-84b1-59f038e90663 16f83d63-e2d2-4c2f-8fd9-9f2c2095eaba [21:53:36] (03PS1) 10Eileen: Do not attempt to calculate contribution_extra for Adyen fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203554 (https://phabricator.wikimedia.org/T399782) [22:05:35] (03merge) 10dwisehaupt: Upgrade civi to 6.8.0 [repos/fundraising-tech/community-civicrm] (labs_build) - 10https://gitlab.wikimedia.org/repos/fundraising-tech/community-civicrm/-/merge_requests/25 [22:17:33] (03PS1) 10Eileen: Move fee disambiguation into AuditMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) [22:28:19] (03PS1) 10Eileen: Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) [22:33:56] (03CR) 10CI reject: [V:04-1] Move fee disambiguation into AuditMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [22:34:06] (03PS1) 10Eileen: Address deprecation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203560 (https://phabricator.wikimedia.org/T399782) [22:35:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Apple Pay donors seeing first & lastname error messages - https://phabricator.wikimedia.org/T407033#11361269 (10MBeat33) One more from today, CID [[ https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=50298665 | 50298665... [22:35:58] (03CR) 10Lars SG: [C:03+2] Move fee disambiguation into AuditMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [22:36:59] (03CR) 10Lars SG: [C:03+2] "Looks good, other than the test fails" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [22:37:28] larssandergreen: that test fail - I've seen that before but more like a time of day fail? [22:38:25] it doesn't fail locally - but it's a different time of day here... [22:43:01] (03PS1) 10Eileen: Add fail information to test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203561 [22:43:22] (03PS2) 10Eileen: Move fee disambiguation into AuditMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) [22:45:06] (03CR) 10CI reject: [V:04-1] Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [23:03:46] (03PS2) 10Eileen: Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) [23:04:45] (03CR) 10Eileen: "thanks for that - it was def a first quick-foray" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201916 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [23:09:17] (03CR) 10Eileen: [C:03+2] "I have a reservation about the use of a . in the settings name - I see with AND with _ are already in use in the file though" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202851 (https://phabricator.wikimedia.org/T404396) (owner: 10Ejegg) [23:10:43] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203555 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [23:19:55] (03CR) 10CI reject: [V:04-1] Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [23:29:57] (03Merged) 10jenkins-bot: Fix typos in upstream matching gifts info [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202851 (https://phabricator.wikimedia.org/T404396) (owner: 10Ejegg) [23:42:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:42:13] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [23:42:13] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [23:42:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 332 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 1 hours - replication_delay is 332, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [23:42:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 338 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 1 hours - replication_delay is 338, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [23:42:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 334 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 41 minutes - replication_delay is 334, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [23:42:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 340 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 41 minutes - replication_delay is 340, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.10%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [23:43:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [23:45:15] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [23:45:19] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [23:46:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [23:46:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [23:47:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:47:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [23:47:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [23:47:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 632 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 1 hours - replication_delay is 632, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [23:47:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 638 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 1 hours - replication_delay is 638, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.00%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [23:47:21] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 635 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 46 minutes - replication_delay is 635, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [23:47:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 641 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 46 minutes - replication_delay is 641, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.12%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [23:48:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [23:50:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [23:50:19] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [23:51:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [23:51:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [23:52:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:52:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [23:52:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [23:52:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 932 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 1 hours - replication_delay is 932, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [23:52:17] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 938 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 1 hours - replication_delay is 938, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.00%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [23:52:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 934 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 51 minutes - replication_delay is 934, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [23:52:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 940 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 51 minutes - replication_delay is 940, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.12%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [23:53:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [23:55:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [23:55:19] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [23:56:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [23:56:15] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [23:57:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:57:13] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [23:57:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [23:57:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1232 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 1 hours - replication_delay is 1232, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [23:57:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1238 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 1 hours - replication_delay is 1238, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 4.00%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [23:57:21] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1234 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 31 keys, up 12 days 56 minutes - replication_delay is 1234, memory use is 259.38M (peak 518.51M, 3.36% of max, fragmentation 1.06%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [23:57:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1240 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 12 days 56 minutes - replication_delay is 1240, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.12%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [23:58:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql