[07:17:49] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2791727 (10Marostegui) dbstore2001 caught up without any problems. This ticket is ready to be closed as soon as @Papaul confirms that the old disks are wiped. [09:12:57] 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#2791833 (10Marostegui) This is now running in db1064, the last server to fix. [09:35:41] 10DBA, 06Labs, 10Tool-Labs-tools-Other: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789420 (10akosiaris) I am the one that responded (partly) to the page by increasing the space on said filesystem. That was done via LVM resizing and XFS grow filesy... [09:38:02] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2791904 (10Marostegui) I have deployed this flag to the following hosts: db1009 (m5 master) db1046 (m4 master - no slaves) db1016 (m1 master) db1001 (m1 slave) ``` root@neodymium:/home/maroste... [10:03:32] 10DBA: Wikidatawiki revision table needs unification - https://phabricator.wikimedia.org/T150644#2791954 (10Marostegui) [11:58:08] 10DBA: Wikidatawiki revision table needs unification - https://phabricator.wikimedia.org/T150644#2792165 (10Marostegui) [11:58:10] 07Blocked-on-schema-change, 10DBA, 05MW-1.28-release-notes, 13Patch-For-Review: Clean up revision UNIQUE indexes - https://phabricator.wikimedia.org/T142725#2792166 (10Marostegui) [13:59:23] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2792364 (10Marostegui) I have started to transfer dewiki to dbstore2001 from db2066 [14:17:10] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2792408 (10chasemp) >>! In T148561#2791429, @Halfak wrote: >> on existing wiki's starting with these as X happens to add t... [14:40:22] 10DBA, 06Labs: Rename old database - https://phabricator.wikimedia.org/T150659#2792479 (10jcrespo) a:03jcrespo [15:41:07] 10DBA, 06Labs: Add visitingwatchers to watchlist_count - https://phabricator.wikimedia.org/T150547#2789248 (10jcrespo) Dispenser, thanks for the separate task. If you are ok with that, I will put T59617 as a blocker of this, not a subtask. The request is reasonable, but this being a very dynamic field it won'... [15:55:19] 10DBA, 06Labs: Reduce watchlist_count threshold - https://phabricator.wikimedia.org/T150548#2789278 (10jcrespo) Dispenser, this is a more complicated request, because it is not about exposing more data in tools, it is about exposing more data in general. Looking at the api, currently the threshold is at "Fe... [15:56:57] 10DBA, 06Labs, 10Wikimedia-Site-requests: Reduce watchlist_count threshold - https://phabricator.wikimedia.org/T150548#2792575 (10jcrespo) [16:19:14] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2792646 (10Marostegui) The transfer finished for both, dewiki and wikidatawiki. I am now importing dewiki's tablespace. [16:35:50] 10DBA, 06Labs: Add visitingwatchers to watchlist_count - https://phabricator.wikimedia.org/T150547#2792687 (10Dispenser) I'm fine with stale data. With the API, visitingwatchers represents 0.005 per watcher checked, in total ~3 - 30 second. Is there a way to get a last modified time like with information_sch... [16:44:18] 10DBA, 06Labs: Add visitingwatchers to watchlist_count - https://phabricator.wikimedia.org/T150547#2792743 (10jcrespo) > Is there a way to get a last modified time like with information_schema? Not now, but we could add such information somewhere. [18:02:54] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2792952 (10jcrespo) Note @Marostegui commented in favor of proxysql before I edited the tasks based on my observations of recent bugs, I would like t... [18:04:47] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2792959 (10Marostegui) As we spoke on hangouts a week ago and after seeing the bug list you posted, I am fine with not going with it to production. M... [18:19:48] 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2793013 (10Marostegui) labsdb1001 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# mysql -hlabsdb1001 -A commonswiki -e "show create table revision\G" *************************** 1. row ********... [18:20:08] 10DBA, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2793017 (10jcrespo) [18:20:11] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2793014 (10jcrespo) 05Open>03Resolved a:03jcrespo > But as I expressed, I would like to still deploy it somewhere in our infra (as we discussed... [19:40:49] hey, random observation, I noticed labsdb1004 doesn't have the prometheus user or prometheus can't otherwise access mysql, known? [19:42:46] I saw it on prometheus, but didn't brought my attention, I supposed it was temporal [19:42:59] it used to work [20:00:44] heh I was surprised too it didn't work [22:25:53] sigh labsdb1004 has warnings for disk space again [23:03:16] godog: T150553 ? [23:03:17] T150553: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553 [23:04:21] looks like it volans [23:05:09] but the folks said that they stopped the tool [23:05:10] let me check [23:07:23] godog: is not the relay log this time [23:09:01] mhh binlogs accumulating away? looks like there's 100MB every 30min [23:11:37] also its master 1005 has 8T for /srv/labsdb so 1004 will get full before anyways [23:12:03] godog: since the reported sizes of alex yesterday in the task I see 2GB less used in /data/labsdb [23:14:01] I think the alarm was still on, it's a warning [23:14:04] *already on [23:14:31] yeah that's true, looks like usage is almost constant [23:16:01] I've nudged it a little bit more with 100G so it gets out of the warning too [23:16:48] 150G that is [23:16:51] lol [23:16:59] ok [23:17:16] and probably make sense to ping the dba folks to see if the old stuff there could be deleted [23:18:12] indeed [23:18:55] TIL icinga will display a short popup with the host status when typing the hostname in the search bar on the left [23:22:50] godog: yeah, I think is since the migration to the new hosts [23:22:51] it's nice [23:25:50] yup, also realtime now! [23:32:11] near-realtime :D