[08:13:48] (CR) Jforrester: [C: +2] Declare arrays before use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [08:25:08] (CR) Jforrester: [C: +2] ""The .git directory is missing from /workspace/src/vendor/wikimedia/smash-pig". That's a new error to me." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [08:48:57] (CR) jerkins-bot: [V: -1] Declare arrays before use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [08:49:37] Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog: FRUEC: Debug large discrepancy in data in initial test run. - https://phabricator.wikimedia.org/T235284 (AndyRussG) I've dug deeper into the Landing Page discrepancy, comparing sequences of log entries... [13:10:30] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Move DonationInterface messages used in banners to WikimediaMessages - https://phabricator.wikimedia.org/T231474 (Ejegg) @pcoombe and the Fundraising banner team would know best. As we're getting close to peak banner season, I'm now doubtful th... [13:14:43] Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, FR-Q2-FY2019-20-cleanup-list, Recurring-Donations, FR-Email: Schedule job for annual totals emails - https://phabricator.wikimedia.org/T234360 (Ejegg) @MBeat33 sure, 3 days instead of 3 weeks would bump us up to 280 / hr, still well below... [13:35:42] Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, FR-Q2-FY2019-20-cleanup-list, Recurring-Donations, FR-Email: Schedule job for annual totals emails - https://phabricator.wikimedia.org/T234360 (MBeat33) Thank you @Ejegg for confirming the accelerated time is not a problem. Good point on... [15:54:49] Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog, FR-PayPal-ExpressCheckout, Patch-For-Review: PayPal recurring audit 10/21/2018 duplicate - https://phabricator.wikimedia.org/T209294 (Ejegg) [16:00:58] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, and 4 others: Ingenico refunds in mismatched ... - https://phabricator.wikimedia.org/T210473 [16:01:45] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 5 others: Ingenico Connect hosted checkouts - using brok... - https://phabricator.wikimedia.org/T210893 [16:02:25] Fundraising Sprint XML ate my homework, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Investigate unsubscribe queue consumer failure - https://phabricator.wikimedia.org/T211846 (Ejegg) [16:02:57] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, Fundraising-Backlog: Adjust codfw payments servers to lo... - https://phabricator.wikimedia.org/T218661 [16:04:18] Fundraising Sprint They Live, Fundraising-Backlog: Ingenico Orphan Slayer missing order status - https://phabricator.wikimedia.org/T205643 (Ejegg) [16:19:18] Fundraising Sprint Sysadmin Kane, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, and 4 others: Annual totals receipt thank you email for recurring donors - https://phabricator.wikimedia.org/T195907 (TSkaff) @Ejegg @CDenes_WMF @MBeat33 @DStrine... [16:20:48] Fundraising Sprint Vestigial tails shoot from the hip, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, Patch-For-Review: Deal with PayPal subscr_payment messages with updated subscr_ids and no indication of the old ones. - https://phabricator.wikimedia.org/T209008 (Ejegg) [16:41:34] thanks for the +2 ejegg [16:41:44] I'll try redeploying again [16:51:43] Fundraising-Backlog, fundraising-tech-ops, observability, Epic: Capture metrics for postfix queue depth both overall and for specific queues and domains - https://phabricator.wikimedia.org/T235804 (Dwisehaupt) Ran across an edge case in the basic version where messages are injected into the queue... [16:52:58] Fundraising Sprint Sysadmin Kane, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, and 4 others: Annual totals receipt thank you email for recurring donors - https://phabricator.wikimedia.org/T195907 (Ejegg) I interpreted @MBeat33's suggested tex... [16:58:44] (CR) Jgleeson: [C: +2] "recheck" [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 (owner: Jgleeson) [17:02:26] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 (owner: Jgleeson) [17:07:03] Fundraising Sprint Sysadmin Kane, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, and 4 others: Annual totals receipt thank you email for recurring donors - https://phabricator.wikimedia.org/T195907 (DStrine) [17:07:46] Fundraising Sprint Sysadmin Kane, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, and 4 others: Annual totals receipt thank you email for recurring donors - https://phabricator.wikimedia.org/T195907 (DStrine) Official email copy is here: https:/... [17:07:53] ejegg: the build is still failing post the update to smash-pig :/ [17:08:21] https://integration.wikimedia.org/ci/job/mwgate-composer-php70-docker/3845/console [17:10:17] ahh I wonder if I need to update the composer.lock file also as it's trying to downgrade the version checked in to the vendor project [17:10:38] - Updating wikimedia/smash-pig (v0.5.10.4 => v0.5.10.3): [17:15:15] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for ebjune - https://phabricator.wikimedia.org/T236710 (Dwisehaupt) [17:25:23] jgleeson: yep, add the composer.lock change and the vendor pointer update to that merge [17:29:22] (PS2) Jgleeson: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 [17:33:17] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 (owner: Jgleeson) [17:50:02] (PS3) Jgleeson: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 [18:19:22] (CR) Jgleeson: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 (owner: Jgleeson) [18:19:25] finally! [18:20:48] (Abandoned) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/432612 (owner: Jgleeson) [18:23:43] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/546225 (owner: Jgleeson) [18:27:53] !log updated paymentswiki from 7bb9f5257e to e28bc54e85 [18:27:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:31:18] (CR) Umherirrender: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [18:34:34] (CR) Umherirrender: [C: +2] "Resubmit" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [18:36:02] (Merged) jenkins-bot: Declare arrays before use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/544239 (owner: Umherirrender) [18:42:42] Fundraising Sprint Sysadmin Kane, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, and 2 others: Civi: create annual $ summary TY email button - https://phabricator.wikimedia.org/T221271 (DStrine) @spatton @CaitVirtue could we get final copy for... [18:47:02] (PS1) Umherirrender: Remove too much arguments for PaymentResult::newFailure [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/546677 [19:04:23] Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (DStrine) Resolved→Open [19:04:48] Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Usual Subscripts, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (DStrine) Open→Resolved [19:06:55] (PS3) Ejegg: End of year summary: align templates with DS proposal [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/545668 (https://phabricator.wikimedia.org/T195907) [19:06:57] (PS2) Ejegg: End of year summary: remove test mode [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546286 (https://phabricator.wikimedia.org/T195907) [19:06:59] (PS2) Ejegg: End of year summary: Fix 'batch' param name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546287 (https://phabricator.wikimedia.org/T195907) [19:07:37] (PS4) Ejegg: End of year summary: add Civi activity [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/540468 (https://phabricator.wikimedia.org/T195907) [19:11:30] (PS4) Ejegg: End of year summary: align templates with DS proposal [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/545668 (https://phabricator.wikimedia.org/T195907) [19:11:32] (PS3) Ejegg: End of year summary: remove test mode [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546286 (https://phabricator.wikimedia.org/T195907) [19:11:34] (PS3) Ejegg: End of year summary: Fix 'batch' param name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546287 (https://phabricator.wikimedia.org/T195907) [19:14:49] (PS5) Ejegg: End of year summary: add Civi activity [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/540468 (https://phabricator.wikimedia.org/T195907) [19:19:41] (CR) Ejegg: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/545955 (https://phabricator.wikimedia.org/T235126) (owner: XenoRyet) [19:24:57] hey ejegg I'm just looking over your eoy summary patches. Should I be able to run `wmf-eoy-calculate-summaries` and `wmf-eoy-send-letters` without a jobId ? it keeps asking me for one [19:25:05] (CR) Ejegg: "Is the continue button not showing for these donors? The original intent was to leave the card selected and show the continue button, givi" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/545955 (https://phabricator.wikimedia.org/T235126) (owner: XenoRyet) [19:25:06] I was kinda assuming it was just gonna send me a ton of emails [19:25:53] jgleeson there's an all-in-one command for that: wmf-eoy-receipts [19:26:18] yeah I tried that also but figured I should break it up as that wanted a job id also [19:26:32] ooh, I must have screwed something up [19:26:35] lemme check [19:26:39] vagrant@mwv:/vagrant/srv/org.wikimedia.civicrm/drupal$ drush wmf-eoy-send-letters [19:26:41] Missing required option: --job_id. See `drush help wmf-eoy-send-letters` for information on [error] [19:26:43] usage. [19:26:51] right, send-letters needs a job id [19:27:04] calculate-summaries should generate a job_id and print it out [19:27:47] I got this [19:27:49] vagrant@mwv:/vagrant/srv/org.wikimedia.civicrm/drupal$ drush wmf-eoy-calculate-summaries [19:27:51] Calculation successful - now you can send the letters with 'drush wmf-eoy-send-letters --job_id=' [19:28:13] (this is on master, I wanted to confirm that worked before trying your patch) [19:28:15] ah, drat, that should have a number at the end [19:28:22] lemme see what I messed up [19:28:37] ejegg: this is on master btw [19:28:47] to save you looking for a red herring on the patch [19:28:58] yep, thanks [19:29:12] oh man [19:29:24] is that was red herring means... something none important? [19:29:36] I've gone my whole life thinking it means something that can't be found [19:29:43] wondering if module_invoke actually returns what I think it does [19:39:16] (PS1) Ejegg: Print Job ID when calculating summaries [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546693 (https://phabricator.wikimedia.org/T195907) [19:39:56] jgleeson: ah, it was a brainfart. Fixed in ^^^ [19:40:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:42:08] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:42:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:45:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:46:53] Fundraising-Backlog: Adapt Druid storage of CentralNotice data following campaign fallback - https://phabricator.wikimedia.org/T236734 (AndyRussG) [19:47:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:47:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:48:24] (CR) Jgleeson: [C: +2] Print Job ID when calculating summaries [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546693 (https://phabricator.wikimedia.org/T195907) (owner: Ejegg) [19:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:50:27] Fundraising-Backlog: Changes coming to Dlocal reporting to help reconciliation - https://phabricator.wikimedia.org/T236524 (DStrine) @EMartin do they have examples for us? It would help if we could look at these in advance. [19:50:29] Fundraising-Backlog: Changes coming to Dlocal reporting to help reconciliation - https://phabricator.wikimedia.org/T236524 (Ejegg) Thanks for the heads-up @EMartin. Can you ask them for examples of the new files? [19:52:10] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:52:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:54:13] (Merged) jenkins-bot: Print Job ID when calculating summaries [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546693 (https://phabricator.wikimedia.org/T195907) (owner: Ejegg) [19:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:57:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [19:57:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:02:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:02:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:03:01] Fundraising Sprint Usual Subscripts, Fundraising-Backlog, FR-Civi-Prospect, Unplanned-Sprint-Work: Cannot add or change anything on the prospect tab - https://phabricator.wikimedia.org/T236096 (Eileenmcnaughton) @NNichols good to close? [20:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:07:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:07:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:12:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:12:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:15:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [20:17:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:17:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:19:49] !log disabled all fundraising scheduled jobs [20:19:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:22:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:22:28] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 40 Threads: 2 Questions: 24 Slow queries: 11 Opens: 561 Flush tables: 1 Open tables: 200 Queries per second avg: 0.600 [20:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:27:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:32:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:37:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [20:42:08] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1482607 Threads: 3 Questions: 109660077 Slow queries: 568417 Opens: 27516903 Flush tables: 1 Open tables: 200 Queries per second avg: 73.964 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:45:12] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 2252729 Threads: 3 Questions: 176507021 Slow queries: 99745 Opens: 44133 Flush tables: 1 Open tables: 200 Queries per second avg: 78.352 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:49:45] Fundraising-Backlog, fundraising-tech-ops, DC-Ops: frdb1001 has suffered a raid event resulting in /dev/sda going read only - https://phabricator.wikimedia.org/T236739 (Dwisehaupt) p:Triage→Unbreak! [21:14:10] (PS1) Ejegg: Add missing args to parent tableInfo fn call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546735 [21:32:01] dstrine: on another topic civi core team seem OK for that meeting [21:34:51] (PS1) MaxSem: PHPUnit 8 compatibility that doesn't break b/c [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/546749 (https://phabricator.wikimedia.org/T192167) [21:38:29] (CR) jerkins-bot: [V: -1] PHPUnit 8 compatibility that doesn't break b/c [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/546749 (https://phabricator.wikimedia.org/T192167) (owner: MaxSem) [21:45:19] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-Paypal, MW-1.35-notes (1.35.0-wmf.2; 2019-10-15): Don't cancel PayPal recurring donations after 3 failed attempts - https://phabricator.wikimedia.org/T235022 (EMartin) Resolved→Open [21:46:12] Fundraising-Backlog: erroneous Paypal subscription termination messaging impacting donor services - https://phabricator.wikimedia.org/T234988 (EMartin) Note: this task was discussed on a call and it spawned the following Phab task here which I reopened. https://phabricator.wikimedia.org/T235022 Conclusio... [21:50:07] (CR) Eileen: [C: +2] Add missing args to parent tableInfo fn call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546735 (owner: Ejegg) [21:51:09] (PS5) Ejegg: Add read only db-connector [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/538373 (owner: Eileen) [21:52:34] (CR) Ejegg: [C: +2] "Really clever! I like how the classifier defaults to WRITE mode if it can't recognize the query as safe." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/538373 (owner: Eileen) [21:58:32] (Merged) jenkins-bot: Add read only db-connector [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/538373 (owner: Eileen) [22:02:10] !log re-enabled basic fundraising jobs (Queue consumers, audit processors, TY mailer) [22:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:02:37] (PS2) Eileen: Add missing args to parent tableInfo fn call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546735 (owner: Ejegg) [23:12:18] PROBLEM - check_mysql on frdb1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [23:14:20] ut-oh ^^^ [23:15:14] oh, that's frdb1001 [23:15:21] we already know that's down [23:15:31] I guess that check just came back on? [23:17:18] PROBLEM - check_mysql on frdb1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [23:18:56] (PS9) Eileen: Move logic for handling Yes-No fields during dedupe from wmf_civicrm to deduper. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/543741 (https://phabricator.wikimedia.org/T235450) [23:18:58] (PS5) Eileen: Add handling for resolving boolean conflict on 'email.on_hold [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/545676 (https://phabricator.wikimedia.org/T221914) [23:19:00] (PS4) Eileen: Improve docs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/545697 [23:19:02] (PS1) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 [23:22:18] PROBLEM - check_mysql on frdb1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [23:22:37] (CR) jerkins-bot: [V: -1] Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 (owner: Eileen) [23:24:26] (PS2) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 [23:27:39] (CR) jerkins-bot: [V: -1] Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 (owner: Eileen) [23:32:18] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5096 [23:32:18] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=467 [critical = 100] [23:33:56] (PS3) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 [23:37:11] (CR) jerkins-bot: [V: -1] Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 (owner: Eileen) [23:37:12] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 682 Threads: 1 Questions: 180531 Slow queries: 267 Opens: 746 Flush tables: 1 Open tables: 575 Queries per second avg: 264.708 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:37:18] RECOVERY - check_rsyslog_backlog on frdb1001 is OK: OK [23:39:52] (PS4) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 [23:43:15] (CR) jerkins-bot: [V: -1] Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 (owner: Eileen) [23:43:46] fundraising-tech-ops: EPIC: migrate fundraising off of Debian Jessie - https://phabricator.wikimedia.org/T185013 (Jgreen) [23:43:48] fundraising-tech-ops: upgrade fundraising databases from Debian Jessie to Stretch - https://phabricator.wikimedia.org/T221003 (Jgreen) Open→Resolved done when bringing frdb1001 back from RAID system crash [23:51:08] (PS5) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 [23:55:14] (CR) jerkins-bot: [V: -1] Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759 (owner: Eileen) [23:56:43] (PS1) Eileen: Phpunit fixes preparatory to upgrading to phpunit 6 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546765 [23:59:10] (PS6) Eileen: Update phpunit version. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/546759