[00:23:06] fundraising-tech-ops: encrypt email traffic between fundraising servers - https://phabricator.wikimedia.org/T170324 (Dwisehaupt) With changes over the last year, we can confirm in the logs that all smtp/smtpd traffic logged on the fundraising servers is using TLS. Most are seeing "untrusted" TLS connections... [00:23:58] db alters on frdb2002 will continue under the screen session. it's chugging along. [00:24:13] i'll be back online to check on it over the weekend. [04:25:39] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3534 [04:30:39] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3834 [04:34:43] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3834 Dwisehaupt backup + virus scan + compression == replication lag. its ok. [04:40:39] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 738188 Threads: 16 Questions: 12775619 Slow queries: 443199 Opens: 24538537 Flush tables: 1 Open tables: 200 Queries per second avg: 17.306 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [17:59:52] table alters still rolling along fine. will check in later. [19:52:23] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Preparing Slave SQL: Yes Seconds Behind Master: (null) [19:57:29] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 94097 Threads: 11 Questions: 894413 Slow queries: 40 Opens: 1430 Flush tables: 1 Open tables: 1124 Queries per second avg: 9.505 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [21:19:48] !log ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY; [21:19:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log