[00:16:58] ejegg: just looking at upgrading the db - I won't be able to turn off alerts & sync will I - I guess I have to wait [00:51:32] eileen: hi! [00:51:39] hey [00:51:59] what's the issue with turning off alerts? [00:52:04] ohh, replication? [00:52:17] well I can't do it myself & I also can't turn off db sync [00:52:40] I'm sitting here with Jeff and Dallas [00:52:42] it takes 30 mins to add the is_template column to civicrm_contribution & then another 30 to add it to log_civicrm_contribuion [00:52:45] ah OK [00:53:09] well I mean I'm OK to do it but I don't want to impinge on your time over there [00:53:58] Jeff says tomorrow during the workshop sessions would be fine [00:54:16] OK - I can do that - when would that be? [00:54:37] We can also separate the column adds from the update - in one of 2 ways [00:54:51] so between 11 and 4 PST [00:54:51] one do them first - then I can do the update 'at will' [00:55:22] or 2, hack the codebase so it doesn't know about those columns until we add tthem [00:55:40] Jeff asks if you can just send a direct message to him and Dallas when you're ready tomorrow. [00:56:28] So Jeff and Dallas just need to stop replication or turn off alerts, and then the civi upgrade script can actually do the alters, right? [00:57:19] Ok, Jeff says just stop repl to one of the backups so we have a restore point in case of disaster and let the other one just stall till it catches up [00:57:20] yep - actually it's possible I might need to get them to delete a couple of cache files - sometimes they seem to be present in an awkward place [00:57:37] okok [00:58:20] what time your time would you be available? [00:58:40] I can do anytime really - it doesn't require much - it's just the add columns are slow [00:59:03] $this->addTask('Add is_template field to civicrm_contribution', 'addColumn', 'civicrm_contribution', 'is_template', [00:59:03] "tinyint(4) DEFAULT '0' COMMENT 'Shows this is a template for recurring contributions.'", FALSE, '5.20.alpha1'); [00:59:03] $this->addTask('Add order_reference field to civicrm_financial_trxn', 'addColumn', 'civicrm_financial_trxn', 'order_reference', [00:59:04] "varchar(255) COMMENT 'Payment Processor external order reference'", FALSE, '5.20.alpha1'); [01:01:03] eileen: so Jeff says 2PM our time tomorrow or 11AM your time would be best for them [01:01:17] ok - lets do that [01:01:42] great! [01:02:12] k, gonna grab food [16:10:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi certificate request: Adri - https://phabricator.wikimedia.org/T244025 (MBeat33) [17:42:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Civi certificate request: Adri - https://phabricator.wikimedia.org/T244025 (Dwisehaupt) a:Dwisehaupt I have regenerated the certificate and it has been sent via email. The password has been sent to her via WhatsApp. [17:46:15] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for avargas - https://phabricator.wikimedia.org/T242037 (Dwisehaupt) Open→Resolved p:Triage→Normal a:Dwisehaupt Regenerated in association with T244025. [21:24:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Civi certificate request: Adri - https://phabricator.wikimedia.org/T244025 (MBeat33) Open→Resolved Thank you @Dwisehaupt we appreciate the speedy help [21:25:38] (Abandoned) Eileen: Fix cleanup on opt-in test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566377 (owner: Eileen) [21:28:50] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/569283 [21:29:10] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/569283 (owner: Eileen) [22:00:41] !log process-control config revision is 2a61da0ace disabled process-control [22:00:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:04:11] Jeff_Green: dwisehaupt I just created an outage doc - still cleaning it up but I think the main thing I need it you guys to stop sync & alerts - https://etherpad.wikimedia.org/p/civiupgrade-2020-01-31 [22:16:40] puppet merge is done. downtime should start kicking in for hosts. [22:17:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6160 5000, pending is 4307 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 10 hours - memory use is 8.39M (peak 8.52M, 0.14% of max, fragmentation 1.33%), connected_slaves is 2, donations is 6, jobs is 0, jobs-adyen is 3, jobs-paypal is 2, payments-init is 655, recurring is 0, refund is 0, unsubscribe is 0 [22:18:57] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6160 5000, pending is 4307 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 10 hours - memory use is 8.39M (peak 8.52M, 0.14% of max, fragmentation 1.33%), connected_slaves is 2, donations is 6, jobs is 0, jobs-adyen is 3, jobs-paypal is 2, payments-init is 655, recurring is 0, refund is 0, unsubscribe is 0 Dwiseha [22:18:57] T242056 [22:20:00] dwisehaupt: ok for me to kick in the upgrade now then? [22:20:14] yes. you should be ok to start. [22:25:43] !log civicrm revision changed from ac730a6bcb to fcc5673ee7, config revision is 2a61da0ace [22:25:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:26:54] ok - it's started trying to add the column [23:00:42] first alter table finished. [23:22:12] PROBLEM - check_load on bismuth is CRITICAL: CRITICAL - load average: 46.35, 37.34, 19.16 [23:25:15] it's onto the log tables now [23:27:12] PROBLEM - check_load on bismuth is CRITICAL: CRITICAL - load average: 46.15, 43.07, 26.69 [23:28:28] that bismuth alert is me. [23:29:40] it's just a bit of netio that is showing up as load but the box is perfectly fine. [23:32:18] PROBLEM - check_load on bismuth is CRITICAL: CRITICAL - load average: 45.64, 45.12, 32.15 [23:37:12] PROBLEM - check_load on bismuth is CRITICAL: CRITICAL - load average: 45.95, 45.59, 35.95 [23:40:17] ACKNOWLEDGEMENT - check_load on bismuth is CRITICAL: CRITICAL - load average: 45.95, 45.59, 35.95 Dwisehaupt Active scan running. [23:58:36] it's on the last query now