[01:15:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /srv 30731MiB (10% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3319MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /tmp 10713MiB (86% inode=98%): /var 19325MiB (72% inode=83%): /var/tmp 422MiB (99% inode=99%): /var/log 16765MiB (63 [01:15:08] 9%): /var/log/audit 422MiB (99% inode=99%): /run/user/33 6392MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:20:10] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /srv 48841MiB (16% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3319MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /tmp 10713MiB (86% inode=98%): /var 19325MiB (72% inode=83%): /var/tmp 422MiB (99% inode=99%): /var/log 16771MiB (63% inode=99% [01:20:10] og/audit 422MiB (99% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:38:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 15.74, 116.58, 75.08 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:43:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 2.74, 43.96, 54.85 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:48:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 19.43, 41.07 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:53:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 0.03, 7.12, 29.73 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:58:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 3.54, 4.46, 22.25 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [02:03:10] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 6.36, 18.24 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [02:18:10] RECOVERY - check_load on franio2002 is OK: LOAD OK - total load average: 3.09, 3.27, 8.43 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [06:17:09] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (2042110) = 94.3% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [06:22:09] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (2042110) = 94.3% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [06:32:09] RECOVERY - check_memory on fransw2001 is OK: OK Memory 87% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [06:32:23] 03Fundraising Sprint: Spaghetti code and makefiles, 06Fundraising-Backlog, 07FR-Audience-Dev, 07fr-current-sprint: Pre-renewal notice to Active Annual recurring donors - https://phabricator.wikimedia.org/T393432#11267534 (10MSuijkerbuijk_WMF) Thank you @Ejegg It looks good now - please confirm that it... [18:32:57] (03PS2) 10Eileen: Declare initial_scheme_transaction_id field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193966 [18:33:11] (03PS11) 10Eileen: And see when it reaches here [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1189995 [18:51:04] (03CR) 10CI reject: [V:04-1] And see when it reaches here [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1189995 (owner: 10Eileen) [20:26:24] (03Abandoned) 10Eileen: Allow file name to be passed in to test runAuditor function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193959 (owner: 10Eileen) [20:30:13] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1195762 [20:32:03] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1195762 (owner: 10Eileen) [20:34:04] !log civicrm upgraded from 385f00d8 to 9393addf [20:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:53:41] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, and 3 others: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#11269631 (10Eileenmcnaughton) [21:01:56] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, and 3 others: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#11269642 (10Eileenmcnaughton) There are some overlapping phabs here - I'm narro... [22:44:36] (03PS1) 10Eileen: Initial handling for additional audit information [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195772 (https://phabricator.wikimedia.org/T152249) [22:46:22] (03PS1) 10Eileen: Additional test cleanup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195774 (https://phabricator.wikimedia.org/T152249) [23:07:05] (03PS3) 10Eileen: Fix attempt to bubble up fee information [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1194015 [23:07:05] (03PS3) 10Eileen: Also bubble up merchant payout transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1195094 [23:07:05] (03PS3) 10Eileen: Also include booking_date from payment report [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1195339 (https://phabricator.wikimedia.org/T396071) [23:07:06] (03PS3) 10Eileen: Align amount fields between payment & settlement reports [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1195343 (https://phabricator.wikimedia.org/T152249) [23:07:07] (03PS4) 10Eileen: Bubble up chargeback reversed [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1195345 (https://phabricator.wikimedia.org/T157456) [23:40:27] (03PS4) 10Eileen: Fix attempt to bubble up fee information [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1194015 [23:41:33] (03PS1) 10Eileen: Add additional steps to verify our batch totals against the file totals [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195776 (https://phabricator.wikimedia.org/T152249) [23:57:32] (03CR) 10CI reject: [V:04-1] Add additional steps to verify our batch totals against the file totals [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195776 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen)