[01:41:31] 10DBA, 06Operations: dbstore2002 , --skip-slave-start and Icinga alerts slave_sql_lag - https://phabricator.wikimedia.org/T142273#2529771 (10Dzahn) [01:42:56] 10DBA, 06Operations: dbstore2002 , --skip-slave-start and Icinga alerts slave_sql_lag - https://phabricator.wikimedia.org/T142273#2529783 (10Dzahn) [08:32:22] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2530193 (10greg) [08:32:31] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2409864 (10greg) [09:03:11] 10DBA, 06Operations, 13Patch-For-Review, 05Prometheus-metrics-monitoring: implement performance_schema for mysql monitoring - https://phabricator.wikimedia.org/T99485#2530248 (10jcrespo) [09:03:14] 10DBA, 06Operations, 10Traffic, 06WMF-Legal, and 2 others: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499#2530249 (10jcrespo) [09:36:17] 10DBA, 06Operations: dbstore2002 , --skip-slave-start and Icinga alerts slave_sql_lag - https://phabricator.wikimedia.org/T142273#2530306 (10jcrespo) First of all; thanks for attending it. It didn't page, which means it was not urgent. Both for being being a dbstore and not being at the primary datacenter. So... [09:41:03] 10DBA, 06Operations: dbstore2002 stopped providing mysql service despite the process being running - https://phabricator.wikimedia.org/T142273#2530308 (10jcrespo) [09:44:45] 10DBA, 06Operations: dbstore2002 stopped providing mysql service despite the process being running - https://phabricator.wikimedia.org/T142273#2530325 (10jcrespo) a:03jcrespo The fact that there was not slave lag after the fact happened probably means mysql server itself was up and replicating without proble... [16:55:09] 10DBA, 06Operations: dbstore2002 stopped providing mysql service despite the process being running - https://phabricator.wikimedia.org/T142273#2530504 (10Dzahn) Thanks for the detailed explanation. No, i did not run any command, it must have started by itself again. [19:00:16] 10DBA, 06Labs, 10Tool-Labs: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530621 (10Giftpflanze) [22:11:30] 10DBA, 06Labs, 10Tool-Labs: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530771 (10Multichill) [22:13:01] 10DBA, 06Labs, 10Tool-Labs: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530783 (10Giftpflanze) [22:13:38] 10DBA, 06Labs, 10Tool-Labs: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530621 (10Giftpflanze) [22:22:00] 10DBA, 06Labs, 10Tool-Labs: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530771 (10valhallasw) This seems to be only affecting c3 (labsdb1003). c2 (labsdb1002) is still offline. ``` valhallasw@tools-bastion-03:~$ echo "SELECT * FROM heartbeat_p.heartbeat... [23:20:53] 10DBA, 06Labs, 10Tool-Labs: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530855 (10yuvipanda) labsdb1003 had crashed, @jcrespo rescued it and I see replag go down now. [23:32:22] 10DBA, 06Labs, 10Tool-Labs: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2530868 (10jcrespo) Both labsdb1001 and labsdb1003 crashed this week due to excessive memory pressure (3 GB swap file). Until we determine if there is a tools or tools using more reso... [23:50:06] 10DBA, 06Labs, 10Tool-Labs: Cannot drop table - https://phabricator.wikimedia.org/T142305#2530889 (10Giftpflanze) 05Open>03Resolved a:03Giftpflanze Seems to work again.