[00:10:49] (PS12) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [00:10:51] (PS1) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [00:12:51] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [00:13:35] I hate you zuul [00:14:10] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [00:16:43] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [00:17:33] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [00:51:47] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [01:04:30] (PS1) Eileen: Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 [01:06:41] (CR) jerkins-bot: [V: -1] Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen) [01:07:27] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen) [01:08:05] AndyRussG: I'm sorry if my answer on your question seemed a bit short last night - it was just a rushed response [01:19:29] (PS2) Eileen: Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 [01:41:27] (PS13) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [01:43:15] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [01:43:37] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [01:45:16] (CR) Eileen: Remove blank address test (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen) [02:50:55] (PS2) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [02:59:14] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [03:20:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1201 [03:25:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1501 [03:30:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774 [03:35:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674 [03:40:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [03:45:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1625 [03:50:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1671 [03:55:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1686 [04:00:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624 [04:04:49] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624 Dwisehaupt looking into replication lag. [04:17:17] (PS3) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [04:23:52] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:40:52] (PS4) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [05:48:04] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:09:15] (PS5) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [06:18:21] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [08:37:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2655 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 74 days 16 hours - memory use is 9.64M (peak 59.11M, 0.18% of max, fragmentation 1.53%), connected_slaves is 2, donations is 45, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1252, payments-init is 138, recurring is 1, refund is 0, unsubscribe is 8 [08:42:15] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 74 days 16 hours - memory use is 6.48M (peak 59.11M, 0.15% of max, fragmentation 1.87%), connected_slaves is 2, donations is 100, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2004, payments-init is 80, pending is 1255, recurring is 0, refund is 0, unsubscribe is 8 [08:58:31] Wikimedia-Fundraising-Banners: QA 'Pretty' variant - https://phabricator.wikimedia.org/T259933 (Pcoombe) [14:10:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 6473258 Threads: 2 Questions: 1377474735 Slow queries: 305732 Opens: 4486741 Flush tables: 8 Open tables: 200 Queries per second avg: 212.794 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 370 [17:53:16] Fundraising-Backlog, Operations, User-Urbanecm, User-dancy, Wiki-Setup (Create): New wiki for fundraising Thank You pages with similar config as donatewiki - https://phabricator.wikimedia.org/T259002 (Urbanecm) >>! In T259002#6366330, @DStrine wrote: > Fishbowl please and thanks! In that cas... [18:09:24] Fundraising-Backlog, Operations, User-Urbanecm, User-dancy, Wiki-Setup (Create): New wiki for fundraising Thank You pages with similar config as donatewiki - https://phabricator.wikimedia.org/T259002 (Urbanecm) >>! In T259002#6364286, @Ladsgroup wrote: > [...] This wiki is a bit special (comp... [18:20:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1227 [18:25:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1345 [18:30:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1496 [18:35:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1660 [18:40:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1818 [18:45:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1654 [18:47:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2203 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 2 hours - memory use is 11.57M (peak 59.11M, 0.18% of max, fragmentation 1.21%), connected_slaves is 2, donations is 64, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1935, payments-init is 106, recurring is 0, refund is 0, unsubscribe is 8 [18:50:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [18:55:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1924 [18:57:15] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 3 hours - memory use is 8.66M (peak 59.11M, 0.17% of max, fragmentation 1.53%), connected_slaves is 2, donations is 72, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2041, payments-init is 102, pending is 1430, recurring is 1, refund is 0, unsubscribe is 9 [19:00:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2085 [19:05:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2246 [19:10:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2407 [19:15:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2476 [19:20:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2638 [19:23:54] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2638 Dwisehaupt truncate causing replication lag. acking as it is known and will complete today. [20:35:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 6496358 Threads: 2 Questions: 1389491019 Slow queries: 306678 Opens: 10565259 Flush tables: 8 Open tables: 200 Queries per second avg: 213.887 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:13:22] (PS1) Pcoombe: Add link to campaign settings log on campaign page [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/619140 (https://phabricator.wikimedia.org/T175841)