[00:10:49] <wikibugs>	 (PS12) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316)
[00:10:51] <wikibugs>	 (PS1) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316)
[00:12:51] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[00:13:35] <eileen>	 I hate you zuul
[00:14:10] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[00:16:43] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[00:17:33] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[00:51:47] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[01:04:30] <wikibugs>	 (PS1) Eileen: Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078
[01:06:41] <wikibugs>	 (CR) jerkins-bot: [V: -1] Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen)
[01:07:27] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen)
[01:08:05] <eileen>	 AndyRussG: I'm sorry if my answer on your question seemed a bit short last night - it was just a  rushed response
[01:19:29] <wikibugs>	 (PS2) Eileen: Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078
[01:41:27] <wikibugs>	 (PS13) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316)
[01:43:15] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[01:43:37] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[01:45:16] <wikibugs>	 (CR) Eileen: Remove blank address test (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 (owner: Eileen)
[02:50:55] <wikibugs>	 (PS2) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316)
[02:59:14] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[03:20:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1201
[03:25:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1501
[03:30:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774
[03:35:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674
[03:40:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766
[03:45:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1625
[03:50:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1671
[03:55:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1686
[04:00:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624
[04:04:49] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624 Dwisehaupt looking into replication lag.
[04:17:17] <wikibugs>	 (PS3) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316)
[04:23:52] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[05:40:52] <wikibugs>	 (PS4) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316)
[05:48:04] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[06:09:15] <wikibugs>	 (PS5) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316)
[06:18:21] <wikibugs>	 (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen)
[08:37:15] <icinga-wm>	 PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2655 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 74 days 16 hours - memory use is 9.64M (peak 59.11M, 0.18% of max, fragmentation 1.53%), connected_slaves is 2, donations is 45, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1252, payments-init is 138, recurring is 1, refund is 0, unsubscribe is 8
[08:42:15] <icinga-wm>	 RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 74 days 16 hours - memory use is 6.48M (peak 59.11M, 0.15% of max, fragmentation 1.87%), connected_slaves is 2, donations is 100, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2004, payments-init is 80, pending is 1255, recurring is 0, refund is 0, unsubscribe is 8
[08:58:31] <wikibugs>	 Wikimedia-Fundraising-Banners: QA 'Pretty' variant - https://phabricator.wikimedia.org/T259933 (Pcoombe)
[14:10:15] <icinga-wm>	 RECOVERY - check_mysql on frdev1001 is OK: Uptime: 6473258 Threads: 2 Questions: 1377474735 Slow queries: 305732 Opens: 4486741 Flush tables: 8 Open tables: 200 Queries per second avg: 212.794 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 370
[17:53:16] <wikibugs>	 Fundraising-Backlog, Operations, User-Urbanecm, User-dancy, Wiki-Setup (Create): New wiki for fundraising Thank You pages with similar config as donatewiki - https://phabricator.wikimedia.org/T259002 (Urbanecm) >>! In T259002#6366330, @DStrine wrote: > Fishbowl please and thanks!  In that cas...
[18:09:24] <wikibugs>	 Fundraising-Backlog, Operations, User-Urbanecm, User-dancy, Wiki-Setup (Create): New wiki for fundraising Thank You pages with similar config as donatewiki - https://phabricator.wikimedia.org/T259002 (Urbanecm) >>! In T259002#6364286, @Ladsgroup wrote: > [...] This wiki is a bit special (comp...
[18:20:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1227
[18:25:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1345
[18:30:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1496
[18:35:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1660
[18:40:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1818
[18:45:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1654
[18:47:15] <icinga-wm>	 PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2203 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 2 hours - memory use is 11.57M (peak 59.11M, 0.18% of max, fragmentation 1.21%), connected_slaves is 2, donations is 64, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1935, payments-init is 106, recurring is 0, refund is 0, unsubscribe is 8
[18:50:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790
[18:55:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1924
[18:57:15] <icinga-wm>	 RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 3 hours - memory use is 8.66M (peak 59.11M, 0.17% of max, fragmentation 1.53%), connected_slaves is 2, donations is 72, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2041, payments-init is 102, pending is 1430, recurring is 1, refund is 0, unsubscribe is 9
[19:00:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2085
[19:05:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2246
[19:10:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2407
[19:15:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2476
[19:20:15] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2638
[19:23:54] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2638 Dwisehaupt truncate causing replication lag. acking as it is known and will complete today.
[20:35:15] <icinga-wm>	 RECOVERY - check_mysql on frdev1001 is OK: Uptime: 6496358 Threads: 2 Questions: 1389491019 Slow queries: 306678 Opens: 10565259 Flush tables: 8 Open tables: 200 Queries per second avg: 213.887 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0
[22:13:22] <wikibugs>	 (PS1) Pcoombe: Add link to campaign settings log on campaign page [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/619140 (https://phabricator.wikimedia.org/T175841)