[07:58:31] PROBLEM - Host frdb1001 is DOWN: PING CRITICAL - Packet loss = 100% [08:05:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:07:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:07:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:10:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:12:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:12:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:15:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:17:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:17:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:20:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:21:18] well this looks bad [08:22:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:22:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:25:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:27:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:27:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:30:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:30:30] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Jeff_Green master is down [08:31:05] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Jeff_Green master is down [08:32:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:32:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:36:13] !log disabled process-control [08:36:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:37] !log took down sidebar fr campaign [08:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:37:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:37:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:42:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:42:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:42:15] RECOVERY - Host frdb1001 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [08:47:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:47:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:52:11] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 2205072 Threads: 2 Questions: 369403654 Slow queries: 1114741 Opens: 1047 Flush tables: 1 Open tables: 729 Queries per second avg: 167.524 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [08:52:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [08:55:15] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 11242303 Threads: 2 Questions: 862049885 Slow queries: 5651631 Opens: 5616 Flush tables: 1 Open tables: 808 Queries per second avg: 76.679 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [08:57:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 2215955 Threads: 2 Questions: 373168267 Slow queries: 327349 Opens: 1866 Flush tables: 1 Open tables: 1332 Queries per second avg: 168.400 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [10:21:06] !log re-enabled process-control [10:21:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:25] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [10:52:25] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [10:57:23] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 15 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [11:02:25] RECOVERY - check_puppetrun on frdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures