[05:55:21] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2651367 (10Marostegui) Thanks for fixing that, I didn't realise that could happen because of db1069. [07:17:01] 10DBA, 06Labs, 10Labs-Infrastructure, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2651434 (10Marostegui) This happened again and I have updated the bug report with Percona and MariaDB with the following information - gdb stacktraces - show engi... [07:18:04] 10DBA, 06Labs, 10Labs-Infrastructure, 07Upstream: db1069: convert user_groups table to InnoDB across all the wikis - https://phabricator.wikimedia.org/T146121#2651436 (10Marostegui) [07:25:31] I reduced yesterday the memory pressure of labsdb1003 https://grafana-admin.wikimedia.org/dashboard/db/server-board?panelId=16&fullscreen&from=1473751506190&to=1474356066190&var-server=labsdb1003&var-network=eth0 [07:26:33] wow nice, what did you do? [08:21:02] I restarted it lowering the buffers [09:10:45] 10DBA, 06Operations, 13Patch-For-Review: Decomission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266762 (10Marostegui) As per our chat, db1019 can be decommissioned [09:19:58] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2651677 (10Marostegui) hitcounter table has been renamed across the board. I will drop it for good in a few days. [10:46:26] 10DBA, 10MediaWiki-JobRunner, 10Wikidata, 07Wikimedia-log-errors: s5-master contention caused (?) by refreshlinksprioritized/addUsagesForPage jobs running for all wikis - https://phabricator.wikimedia.org/T146079#2651823 (10jcrespo) The errors have slowed down to the point they disappeared. I will close th... [10:47:43] ^check a ticket I wrote, marostegui yesterday because I detected a strange pattern [10:47:52] it is not happening anymore [10:48:23] checking [10:48:38] so that you do the same in the future [10:50:14] Oh, interesting [10:51:14] good catch [11:39:02] do you hate me already? :-) [11:39:38] haha no! because I get to practice and polish the commits! [12:40:14] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2651952 (10Marostegui) This server has been running for 24h with no issues so far, reported. I would like to pool it in tomorrow with some weight (not much) to see how it starts coping with production traffic. Any though... [12:40:47] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2651953 (10jcrespo) +1 [12:57:50] 10DBA, 06Labs, 10Labs-Infrastructure, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2651977 (10Marostegui) MariaDB just answered: // Thanks. Lets wait and see what Percona guys come up with, TokuDB is their area of expertise. If they claim it's... [13:49:37] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2652106 (10MoritzMuehlenhoff) Yeah, let's repool and check whether it happens again. [14:07:54] 10DBA, 06Labs, 10Labs-Infrastructure, 05Security: MySQL password has been deployed in clear text to labtestweb2001, potentially other hosts - https://phabricator.wikimedia.org/T146146#2652122 (10jcrespo) [14:10:26] 10DBA, 06Labs, 10Labs-Infrastructure, 05Security: MySQL password has been deployed in clear text to labtestweb2001, potentially other hosts - https://phabricator.wikimedia.org/T146146#2652137 (10jcrespo) [16:31:01] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 2 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2652512 (10Addshore) [18:46:20] 10DBA, 10MediaWiki-JobRunner, 10Wikidata, 07Wikimedia-log-errors: s5-master contention caused (?) by refreshlinksprioritized/addUsagesForPage jobs running for all wikis - https://phabricator.wikimedia.org/T146079#2653147 (10jcrespo) 05Open>03Resolved a:03jcrespo There is still some spikes here and th... [22:19:18] 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, 06Release-Engineering-Team, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2654239 (10greg) For the task at hand, I've added https://wikitech.wikim... [22:30:27] 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, 06Release-Engineering-Team, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2654312 (10greg) Ok, emailed. Resolving. Thanks @jcrespo for the sugges... [22:30:36] 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, 06Release-Engineering-Team, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2654313 (10greg) a:03greg [22:31:14] 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, 06Release-Engineering-Team, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2606542 (10greg) 05Open>03Resolved p:05Triage>03Normal [23:15:49] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2654510 (10dduvall) The migration to `deployment-db03` and `deployme... [23:22:23] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2654516 (10dduvall)