[00:01:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [00:02:13] PROBLEM - check_mysql on frdb1003 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [00:11:26] dwisehaupt: Jeff_Green looks like it's finishing up [00:11:38] yay! [00:12:52] hmm it terminated 'abnormally' - but just on refreshing caches & it;'s a known thing- if you re-enable the UI I'll check it out on there [00:13:34] ok, Dallas will have it out of maint mode in a minute [00:16:33] ok. puppet changes done and pushed out. you should be good. [00:17:13] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1341 [00:20:41] all looks good [00:22:13] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1641 [00:22:57] so restarting jobs now [00:25:30] great [00:25:52] actually - we should maybe reload triggers while we are down [00:26:06] it's really administrative this time but... [00:27:15] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1941 [00:30:06] (PS1) Eileen: Update triggers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/569290 [00:32:33] ejegg|semi: are you there to approve the triggers ^^ [00:33:00] i can see him. [00:33:19] the advantages of being there in person :-) [00:33:23] we are staring at him to make him notice us. :) [00:34:18] he's logging in [00:34:23] hi eileen! [00:34:25] I'll take a look [00:35:09] cool - I think it wasn't updated in git when we excluded large donors so that is in [00:35:33] I tested a few merges & Jim merged to James - nice [00:39:32] (CR) Ejegg: [C: +2] Update triggers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/569290 (owner: Eileen) [00:40:14] Looks good eileen! What was that 'civicrm_persistent' table used for? [00:40:28] yeah good question actually! [00:41:04] the upgrade task said $this->addTask('Clean up unused table "civicrm_persistent"', 'dropTableIfEmpty', 'civicrm_persistent'); [00:41:19] huh, ok [00:43:12] I find zuul often seems to fail to recheck & I have to trigger it with 'recheck' [00:44:03] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/569290 (owner: Eileen) [00:45:34] (Merged) jenkins-bot: Update triggers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/569290 (owner: Eileen) [00:46:37] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/569291 [00:46:43] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/569291 (owner: Eileen) [00:47:15] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 5808841 Threads: 1 Questions: 1175292926 Slow queries: 2247035 Opens: 2735 Flush tables: 1 Open tables: 707 Queries per second avg: 202.328 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:50:19] !log civicrm revision changed from fcc5673ee7 to ee9edf8137, config revision is 2a61da0ace [00:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:03:49] !log process-control config revision is c3c8bde761 [01:03:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:03:56] jobs back on [01:05:02] dwisehaupt: you / Jeff_Green have done 22 & 23 in this ? https://etherpad.wikimedia.org/p/civiupgrade-2020-01-31 [01:06:11] not yet. i'll do 23 now. the alerts we'll expire after they catch up. [01:07:58] ok. replication started back up. [01:08:36] (PS1) Ejegg: Better PAN validation for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/569294 (https://phabricator.wikimedia.org/T234342) [01:12:13] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 13 hours - memory use is 2.22M (peak 48.16M, 0.06% of max, fragmentation 2.32%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 245, payments-init is 134, pending is 2, recurring is 3, refund is 0, unsubscribe is 1 [01:41:22] Fundraising Sprint Byzantine Empire Strikes Back, Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (jgleeson) Hmm my first attempt at a similar query to pull out the affected paypal donations isn't working P10296. I added two addition... [01:47:15] RECOVERY - check_load on bismuth is OK: OK - load average: 0.04, 0.07, 4.02 [02:06:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7878 [02:07:13] PROBLEM - check_mysql on frdb1003 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9810 [02:11:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8178 [02:12:13] PROBLEM - check_mysql on frdb1003 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8238 [02:56:13] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1429094 Threads: 1 Questions: 73070806 Slow queries: 916 Opens: 2459 Flush tables: 1 Open tables: 603 Queries per second avg: 51.130 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [03:17:13] RECOVERY - check_mysql on frdb1003 is OK: Uptime: 4963443 Threads: 1 Questions: 897022234 Slow queries: 3212 Opens: 14644 Flush tables: 13 Open tables: 933 Queries per second avg: 180.725 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0