[00:07:39] (03PS5) 10Eileen: Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [00:07:39] (03PS1) 10Eileen: Remove some calls to civicrm_initialize [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153403 (https://phabricator.wikimedia.org/T276272) [00:10:55] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1153405 [00:11:27] (03CR) 10CI reject: [V:04-1] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1153405 (owner: 10Eileen) [00:18:10] (03PS1) 10Eileen: Update vendor (sha change on omnimail) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1153406 [00:18:21] (03CR) 10Eileen: [C:03+2] Update vendor (sha change on omnimail) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1153406 (owner: 10Eileen) [00:18:43] (03PS2) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1153405 [00:23:37] (03CR) 10CI reject: [V:04-1] Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [00:27:19] ok latest fail is weird - but I am investigating.... [00:29:52] oh wow - still being added by offline2civicrm.install [00:35:45] (03Merged) 10jenkins-bot: Update vendor (sha change on omnimail) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1153406 (owner: 10Eileen) [00:36:30] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1153405 (owner: 10Eileen) [00:38:03] !log civicrm upgraded from 8eb67a94 to 22171c0b [00:38:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:40:46] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Problem upgrading CiviCRM with cv - https://phabricator.wikimedia.org/T395977 (10Eileenmcnaughton) 03NEW [00:44:31] 14Fundraising Sprint: didAnyoneTryThis(), 14Fundraising Sprint: Hot Pot!, 06Fundraising-Backlog: Create new Benevity import - https://phabricator.wikimedia.org/T359219#10882362 (10Eileenmcnaughton) @MDemosWMF I ran the update on live - so cross fingers things are looking good now.... [01:00:25] (03PS2) 10Eileen: Remove some calls to civicrm_initialize [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153403 (https://phabricator.wikimedia.org/T276272) [01:00:25] (03PS6) 10Eileen: Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [01:00:26] (03PS1) 10Eileen: Move CustomField creation from module install to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153412 (https://phabricator.wikimedia.org/T276272) [01:01:19] (03PS1) 10Eileen: Remove artifact of defunct civibank import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153413 [01:03:20] (03CR) 10Eileen: [V:03+2 C:03+2] Actually have wmf not wmff [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1153388 (owner: 10Cstone) [01:04:30] (03CR) 10CI reject: [V:04-1] Move CustomField creation from module install to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153412 (https://phabricator.wikimedia.org/T276272) (owner: 10Eileen) [01:04:32] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install frnetmon1002, pay-lb1001, pay-lb1002 - https://phabricator.wikimedia.org/T369565#10882367 (10Dwisehaupt) @VRiley-WMF Thanks. I have rebuilt both of the boxes and things are looking better. pay-lb1001 shows both interfaces up and bo... [01:04:45] (03CR) 10CI reject: [V:04-1] Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [01:05:36] (03CR) 10CI reject: [V:04-1] Remove artifact of defunct civibank import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153413 (owner: 10Eileen) [01:06:58] (03PS2) 10Eileen: Move CustomField creation from module install to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153412 (https://phabricator.wikimedia.org/T276272) [01:06:59] (03PS7) 10Eileen: Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [01:06:59] (03PS2) 10Eileen: Remove artifact of defunct civibank import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153413 [01:27:08] cstone: I'm trying to figure out where my path is set - is yours also set to this on the civicrm docker instance - or is it something I've done myself? [01:27:08] docker$ whereis phpunit [01:27:08] phpunit: /srv/civi-sites/wmff/vendor/phpunit/phpunit/phpunit [01:27:08] - [01:28:05] oh - docker-compose.override.yml - my doing [01:42:30] (03CR) 10Eileen: [C:03+2] Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [01:43:07] woohoo - it passed !! (won't merge until dependencies are +2 but I gave +2 to Elliot's patch [01:43:28] woohoo! [01:44:09] will try to check out those dependencies shortly [02:50:49] (03PS1) 10Eileen: Fix for tests not always firing, load order dependent [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153422 [03:11:51] (03PS3) 10Cstone: WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [03:12:18] (03CR) 10CI reject: [V:04-1] WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) (owner: 10Cstone) [03:12:41] (03PS4) 10Cstone: WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [03:13:07] (03CR) 10CI reject: [V:04-1] WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) (owner: 10Cstone) [03:15:44] (03PS5) 10Cstone: WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [03:16:14] (03CR) 10CI reject: [V:04-1] WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) (owner: 10Cstone) [03:29:13] (03PS2) 10Eileen: Set default job to avoid cross contamination [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153361 (https://phabricator.wikimedia.org/T298381) [03:49:58] (03PS1) 10Eileen: Only retrieve email & recipient_id for suppression list [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153424 (https://phabricator.wikimedia.org/T298381) [03:56:41] (03PS2) 10Eileen: Only retrieve fields present in master suppression DB for suppression list [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153424 (https://phabricator.wikimedia.org/T298381) [04:24:40] (03PS6) 10Cstone: WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [04:25:06] (03CR) 10CI reject: [V:04-1] WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) (owner: 10Cstone) [04:27:45] (03PS7) 10Cstone: WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [04:28:12] (03CR) 10CI reject: [V:04-1] WIP: Name gravy reports based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) (owner: 10Cstone) [04:29:13] (03PS8) 10Cstone: Name gravy report csvs based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [09:27:07] 06Fundraising-Backlog: Removing Data Axle CE fields from Civi - License Expiring - https://phabricator.wikimedia.org/T395961#10883011 (10RLewis) @MDemosWMF - we were going to use some of these fields to help screen donors for inclusion on our MGFE appeals for FY25-26. For FY24-25, we used them and the upgrade as... [09:28:56] 06Fundraising-Backlog: Removing Data Axle CE fields from Civi - License Expiring - https://phabricator.wikimedia.org/T395961#10883017 (10RLewis) @MDemosWMF - also do you have any information/field the Faircom data will include? [12:23:30] (03PS1) 10Jgleeson: Add Slovak translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153606 (https://phabricator.wikimedia.org/T394932) [12:53:54] (03PS3) 10Jgleeson: Add Norwegian translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1152828 (https://phabricator.wikimedia.org/T394932) [12:53:54] (03PS3) 10Jgleeson: Add Polish translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1152829 (https://phabricator.wikimedia.org/T394932) [12:53:54] (03PS2) 10Jgleeson: Add Portuguese translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153111 (https://phabricator.wikimedia.org/T394932) [12:53:55] (03PS2) 10Jgleeson: Add Romanian translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153112 (https://phabricator.wikimedia.org/T394932) [12:53:56] (03PS2) 10Jgleeson: Add Russian translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153113 (https://phabricator.wikimedia.org/T394932) [12:53:58] (03PS2) 10Jgleeson: Add Swedish translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153114 (https://phabricator.wikimedia.org/T394932) [12:54:02] (03PS2) 10Jgleeson: Add Ukrainian translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153115 (https://phabricator.wikimedia.org/T394932) [12:54:06] (03PS2) 10Jgleeson: Add Slovak translation for SEPA mandate notice. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1153606 (https://phabricator.wikimedia.org/T394932) [13:36:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9014 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [13:41:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3217 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [13:41:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9450 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [13:42:58] 06Fundraising-Backlog, 06FR-donorrelations: Apple Pay donation assigned to wrong contact in CiviCRM despite correct name in Gravy and Adyen - https://phabricator.wikimedia.org/T396026 (10SHust) 03NEW [13:46:10] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 664838 Threads: 4 Questions: 123530293 Slow queries: 315 Opens: 2576 Open tables: 1329 Queries per second avg: 185.805 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [13:46:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4424 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [13:51:12] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 506668 Threads: 4 Questions: 97966768 Slow queries: 236 Opens: 2272 Open tables: 1310 Queries per second avg: 193.354 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [13:52:52] (03PS4) 10Jgleeson: Gravy: inject mappers & validators via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149494 (https://phabricator.wikimedia.org/T393958) [13:52:52] (03PS4) 10Jgleeson: Gravy: update BT/RedirectPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149495 (https://phabricator.wikimedia.org/T393958) [13:52:53] (03PS4) 10Jgleeson: Gravy: update ApplePayPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149496 (https://phabricator.wikimedia.org/T393958) [13:52:53] (03PS4) 10Jgleeson: Gravy: update GooglePayPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149497 (https://phabricator.wikimedia.org/T393958) [13:52:54] (03PS5) 10Jgleeson: Gravy: update DirectDepositPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149498 (https://phabricator.wikimedia.org/T393958) [13:52:56] (03PS4) 10Jgleeson: Gravy: update PayPalPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149499 (https://phabricator.wikimedia.org/T393958) [13:53:00] (03PS4) 10Jgleeson: Gravy: update Venmo RedirectPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149500 (https://phabricator.wikimedia.org/T393958) [13:53:04] (03PS5) 10Jgleeson: Gravy: update cash RedirectPaymentProvider to use DI via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149501 (https://phabricator.wikimedia.org/T393958) [14:23:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16048 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:26:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Trouble voiding transaction at Gr4vy - https://phabricator.wikimedia.org/T394090#10884215 (10AMJohnson) As @MBeat33 is OOO for visibility just sharing the response from Gr4vy to the [[https://wikimedia.slack.com/archives/C070F1DVBRN... [14:28:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16348 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:32:50] (03PS1) 10Damilare Adedoyin: Move common fields validation in Redirect PaymentProvider [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153640 [14:33:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16648 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:34:19] (03PS3) 10Damilare Adedoyin: Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 [14:38:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16948 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:43:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17248 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:48:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17548 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:53:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15355 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:58:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12317 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [15:03:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6342 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [15:07:05] 06Fundraising-Backlog, 06FR-donorrelations: Unable to load 'change log' - https://phabricator.wikimedia.org/T396040 (10SHust) 03NEW [15:07:55] (03PS2) 10Damilare Adedoyin: Move common fields validation in Redirect PaymentProvider [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153640 [15:07:55] (03PS4) 10Damilare Adedoyin: Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 [15:08:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 660582 Threads: 4 Questions: 123010064 Slow queries: 340 Opens: 2558 Open tables: 1327 Queries per second avg: 186.214 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [15:08:31] (03CR) 10CI reject: [V:04-1] Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 (owner: 10Damilare Adedoyin) [15:17:47] (03PS5) 10Damilare Adedoyin: Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 [15:18:18] (03CR) 10CI reject: [V:04-1] Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 (owner: 10Damilare Adedoyin) [15:19:03] (03PS6) 10Damilare Adedoyin: Get payment method config and required fields from Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151739 [16:01:15] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 118.83, 88.20, 46.72 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:02:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 125.08, 98.86, 55.02 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:02:13] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 123.37, 97.94, 54.49 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:06:15] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 127.39, 111.68, 68.30 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:07:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 129.09, 116.37, 74.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:07:11] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 124.09, 114.87, 73.93 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:10:07] (03PS9) 10Cstone: Name gravy report csvs based on their name [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1147067 (https://phabricator.wikimedia.org/T394132) [16:11:15] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 130.62, 122.39, 84.77 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:12:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 128.65, 121.81, 88.27 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:12:15] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 131.18, 123.39, 88.75 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:16:15] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 129.19, 125.47, 96.40 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:17:11] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 125.67, 124.14, 98.52 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:17:15] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 124.27, 123.82, 98.64 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:21:15] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 126.90, 125.87, 104.53 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:22:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 124.62, 126.33, 106.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:22:13] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 120.45, 122.20, 104.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:26:13] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 26.08, 84.35, 94.72 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:27:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 15.29, 73.65, 91.50 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:27:11] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 9.65, 68.90, 88.87 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:31:13] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 11.50, 36.31, 70.71 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:32:09] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 7.20, 30.40, 67.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:32:11] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 7.24, 30.16, 66.48 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:37:11] RECOVERY - check_load on fransw2002 is OK: LOAD OK - total load average: 2.07, 12.70, 48.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [16:41:15] RECOVERY - check_load on fransw2003 is OK: LOAD OK - total load average: 0.56, 6.40, 38.09 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [16:42:07] RECOVERY - check_load on fransw2001 is OK: LOAD OK - total load average: 1.22, 7.89, 37.74 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [16:53:46] (03PS1) 10Damilare Adedoyin: Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 [16:56:14] (03PS2) 10Damilare Adedoyin: Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 [17:00:34] (03CR) 10Cstone: [C:03+2] Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 (owner: 10Damilare Adedoyin) [17:04:10] (03PS3) 10Damilare Adedoyin: Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 [17:08:28] (03CR) 10Cstone: [C:03+2] Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 (owner: 10Damilare Adedoyin) [17:08:59] (03Merged) 10jenkins-bot: Suppress buyer message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153659 (owner: 10Damilare Adedoyin) [17:54:09] 06Fundraising-Backlog: Removing Data Axle CE fields from Civi - License Expiring - https://phabricator.wikimedia.org/T395961#10885316 (10MDemosWMF) @RLewis We haven't gotten that far yet, I think we will be coordinating that with Faircom/VeraData and could try and find the same or similar data fields @MSuijkerbu... [17:58:05] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1152299 (owner: 10Jgleeson) [17:58:56] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1152299 (owner: 10Jgleeson) [18:13:18] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153672 [18:13:56] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153672 (owner: 10Damilare Adedoyin) [18:14:21] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153672 (owner: 10Damilare Adedoyin) [18:15:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Unable to load 'change log' - https://phabricator.wikimedia.org/T396040#10885392 (10AKanji-WMF) Hi @Eileenmcnaughton - putting this high chaos item from @SHust on your radar - TY [18:16:16] !log SmashPig upgraded from a99f2265 to d08693e5 [18:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:21:58] 03Fundraising Sprint: Kimchee Pancake, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Import templates missing / not saving new - https://phabricator.wikimedia.org/T394929#10885405 (10MDemosWMF) Yes, thanks for checking! @Eileenmcnaughton [18:25:02] 14Fundraising Sprint: didAnyoneTryThis(), 14Fundraising Sprint: Hot Pot!, 06Fundraising-Backlog: Create new Benevity import - https://phabricator.wikimedia.org/T359219#10885423 (10MDemosWMF) @Eileenmcnaughton Thanks! I will do some spot checking on prod and report back. [18:55:30] 10fundraising-tech-ops, 10BDC-Implementation, 10FR-Tech-Analytics: Create mysql event listener to save trino query history - https://phabricator.wikimedia.org/T395338#10885480 (10AStein-WMF) the event listener has been implemented. there's a new `trino` database in mariadb. the `trino_queries` table stores t... [18:55:33] 10fundraising-tech-ops, 10BDC-Implementation, 10FR-Tech-Analytics: Create mysql event listener to save trino query history - https://phabricator.wikimedia.org/T395338#10885481 (10AStein-WMF) 05Open→03Resolved [18:59:42] (03CR) 10Ejegg: [C:03+2] Remove some calls to civicrm_initialize [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153403 (https://phabricator.wikimedia.org/T276272) (owner: 10Eileen) [19:02:47] (03CR) 10Ejegg: [C:03+2] Move CustomField creation from module install to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153412 (https://phabricator.wikimedia.org/T276272) (owner: 10Eileen) [19:02:58] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate FR DMARC parser and database to FR analytics - https://phabricator.wikimedia.org/T394724#10885495 (10Jgreen) [19:03:04] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate FR DMARC parser and database to FR analytics - https://phabricator.wikimedia.org/T394724#10885498 (10Jgreen) [19:03:28] (03CR) 10Ejegg: [C:03+2] Remove artifact of defunct civibank import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153413 (owner: 10Eileen) [19:03:52] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate FR DMARC parser and database to FR analytics - https://phabricator.wikimedia.org/T394724#10885500 (10Jgreen) [19:14:37] (03PS1) 10Ejegg: Update default log tag for Gravy to Smashpig-Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153682 (https://phabricator.wikimedia.org/T395706) [19:15:01] (03PS2) 10Ejegg: Update default log tag for Gravy to SmashPig-Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153682 (https://phabricator.wikimedia.org/T395706) [19:18:07] (03Merged) 10jenkins-bot: Remove some calls to civicrm_initialize [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153403 (https://phabricator.wikimedia.org/T276272) (owner: 10Eileen) [19:18:25] 06Fundraising-Backlog: Searchkit build - Donors from Engage import who need TYLs - https://phabricator.wikimedia.org/T396068 (10MDemosWMF) 03NEW [19:22:24] (03Merged) 10jenkins-bot: Move CustomField creation from module install to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153412 (https://phabricator.wikimedia.org/T276272) (owner: 10Eileen) [19:22:25] (03Merged) 10jenkins-bot: Run tests against standalone not D7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1152793 (https://phabricator.wikimedia.org/T276272) (owner: 10Ejegg) [19:22:27] (03Merged) 10jenkins-bot: Remove artifact of defunct civibank import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153413 (owner: 10Eileen) [19:29:15] 06Fundraising-Backlog, 10fundraising-tech-ops, 13Patch-For-Review: Can we get more MariaDB deadlock tools? - https://phabricator.wikimedia.org/T392396#10885632 (10Jgreen) @Eileenmcnaughton I think you should already have access to "show engine innodb status" -- the grant should be "process" and all fr-tech u... [19:57:33] 14Fundraising Sprint: didAnyoneTryThis(), 14Fundraising Sprint: Hot Pot!, 06Fundraising-Backlog: Create new Benevity import - https://phabricator.wikimedia.org/T359219#10885696 (10MDemosWMF) I made that change to the benevity import template, ran a test, and it looks like it is now allocating the right s... [20:00:09] (03PS1) 10Ejegg: WIP Gravy: inject mappers & validators via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153695 (https://phabricator.wikimedia.org/T393958) [20:00:35] (03CR) 10CI reject: [V:04-1] WIP Gravy: inject mappers & validators via config [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153695 (https://phabricator.wikimedia.org/T393958) (owner: 10Ejegg) [20:09:55] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071 (10Eileenmcnaughton) 03NEW [20:23:44] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#10885782 (10Eileenmcnaughton) I think we should get hold of the data for a particular reconciliation date & see if we can get the date [20:46:48] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install frnetmon1002, pay-lb1001, pay-lb1002 - https://phabricator.wikimedia.org/T369565#10885828 (10VRiley-WMF) I have reseated the cable from the switch end for pay-lb1002. Would you be able to check to see if it's there? In Netbox, it d... [20:54:01] 06Fundraising-Backlog, 10fundraising-tech-ops, 13Patch-For-Review: Can we get more MariaDB deadlock tools? - https://phabricator.wikimedia.org/T392396#10885863 (10Eileenmcnaughton) @Jgreen when I log into mysql on prod I do so 'as if' I am the web user. a couple of notes - 1) I was trying to capture extra... [21:09:51] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#10885906 (10Eileenmcnaughton) Need to store fees, not ignore any transaction, [21:14:02] 03Fundraising Sprint: Kimchee Pancake, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Import templates missing / not saving new - https://phabricator.wikimedia.org/T394929#10885931 (10MDemosWMF) @Eileenmcnaughton Oops spoke to soon. I'm spot checking some of the imports and the one you brought back... [21:54:12] (03CR) 10Cstone: [C:03+2] Fix for tests not always firing, load order dependent [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153422 (owner: 10Eileen) [21:57:48] (03PS1) 10Damilare Adedoyin: Fix state code causing failmails in listener [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153726 [21:58:51] damilare: is that for the failmail we just got [21:58:52] cstone: looks like a patch in the deploy to fix the earlier errors is causing another failmail in the gravy listener [21:58:58] yepp [21:59:02] hah i was just looking at that [21:59:27] (03CR) 10CI reject: [V:04-1] Fix state code causing failmails in listener [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153726 (owner: 10Damilare Adedoyin) [21:59:31] state_code melting my brain [22:00:20] yea it's comming as null in the transaction detail and str_replace does not like that [22:00:51] (03PS2) 10Damilare Adedoyin: Fix state code causing failmails in listener [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153726 [22:01:02] spacing issues [22:01:50] ok that passed CI now cstone [22:02:01] thanks damilare ! [22:02:37] (03CR) 10Cstone: [C:03+2] Fix state code causing failmails in listener [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153726 (owner: 10Damilare Adedoyin) [22:13:04] (03Merged) 10jenkins-bot: Fix for tests not always firing, load order dependent [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1153422 (owner: 10Eileen) [22:13:05] (03Merged) 10jenkins-bot: Fix state code causing failmails in listener [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1153726 (owner: 10Damilare Adedoyin) [22:13:25] gonna get that deployed [22:15:24] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153731 [22:16:37] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153731 (owner: 10Damilare Adedoyin) [22:17:03] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1153731 (owner: 10Damilare Adedoyin) [22:18:15] !log SmashPig upgraded from d08693e5 to 3222a1f3 [22:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:26:21] PROBLEM - check_apache2 on frpig1002 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_apache2 [22:30:59] looks like that error has been resolved [22:31:09] PROBLEM - check_apache2 on frpig1002 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_apache2 [22:36:21] PROBLEM - check_apache2 on frpig1002 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_apache2 [22:41:11] RECOVERY - check_apache2 on frpig1002 is OK: PROCS OK: 103 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_apache2 [22:59:24] (03PS1) 10Eileen: Remove some fs [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1153734 [23:09:01] (03PS1) 10Eileen: Remove some fs [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1153735 [23:37:26] 14Fundraising Sprint: didAnyoneTryThis(), 03Fundraising Sprint: Kimchee Pancake, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Configure the necessary apache/nginx portions for civi standalone testing on staging - https://phabricator.wikimedia.org/T396084 (10Dwisehaupt) 03NEW