[05:52:26] 10DBA, 10Patch-For-Review: Test tendril events on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) I just realised that dbstore hosts do not use TokuDB anymore. The only tables we have with tokudb are on tendril and they are: ` +-----------------------+--------------+ | table_name |... [05:54:54] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) [06:25:06] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10MoritzMuehlenhoff) >>! In T248957#6015572, @Marostegui wrote: > Looks like that for the hosts that will require TokuDB (tendril and analytics dbstore) we need to find a way to use jemalloc on start... [06:27:30] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) Maybe we can start by purging `global_log*` tables more often to reduce their size: ` root@db1115.eqiad.wmnet[tendril]> show create event tendril_purge_global_status_log_5m\G ***********... [06:29:11] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) >>! In T248957#6017471, @MoritzMuehlenhoff wrote: >>>! In T248957#6015572, @Marostegui wrote: >> Looks like that for the hosts that will require TokuDB (tendril and analytics dbstore) we... [07:27:53] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10jcrespo) Yep, this is known, tendril doesn't work on buster anyway, so it is a non-issue. [07:28:48] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) >>! In T248957#6017559, @jcrespo wrote: > Yep, this is known, tendril doesn't work on buster anyway, so it is a non-issue. Non-issue? Key tables for tendril are on TokuDB [07:42:23] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10jcrespo) > Non-issue? Key tables for tendril are on TokuDB Yes, the format of the tables doesn't matter if the application cannot even run on buster- see db2093. I sent an email to you and mark men... [07:43:52] 10DBA, 10Patch-For-Review: Test tendril events and tokudb on 10.4 - https://phabricator.wikimedia.org/T248957 (10Marostegui) 05Open→03Declined Ok then [07:43:54] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10Marostegui) [08:01:17] 10DBA: Convert Tendril TokuDB tables to InnoDB - https://phabricator.wikimedia.org/T249085 (10Marostegui) [08:01:45] 10DBA: Convert Tendril TokuDB tables to InnoDB - https://phabricator.wikimedia.org/T249085 (10Marostegui) 05Open→03Stalled p:05Triage→03Medium [08:46:27] 10DBA: Decommission dbproxy1010.eqiad.wmnet - https://phabricator.wikimedia.org/T248944 (10Marostegui) Stopped haproxy on dbproxy1010 - there were no active connections: ` root@cumin1001:/home/marostegui# host dbproxy1010 dbproxy1010.eqiad.wmnet has address 10.64.37.14 root@cumin1001:/home/marostegui# for i in l... [08:47:04] 10DBA: Decommission dbproxy1010.eqiad.wmnet - https://phabricator.wikimedia.org/T248944 (10Marostegui) [09:14:38] what was db2093? tokudb, mariadb, our package? [09:14:46] tendril? [09:15:06] probably toku and the whole mess with the plugin, the different basedir etc [09:15:18] I am going to push https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/585115/ and then will downgrade the package [09:15:52] downgrade to 10.1, 10.4 -1 ? [09:16:28] 10.4 but package with no toku [09:16:33] +1 [09:17:57] thanks! [09:18:27] I think is ok with the >= because on first reinstall we may want to keep package [09:18:33] and we don't know what will come after that [09:18:42] it is ok with the == too [09:32:59] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [10:46:43] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [11:31:31] I just saw a huge drop on db activity (enwiki) yesterday from 1-1:30 UTC [11:31:39] well, technically, today [11:32:21] doesn't show on all graphs, so probably a monitoring issue [12:08:07] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) s8 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1111 [] db1109 [] db1104 [] db1101 []... [14:35:38] 10DBA, 10Operations, 10ops-codfw: (Need by: TBD) codfw: rack/setup/install backup2002/array backup2002-array1 - https://phabricator.wikimedia.org/T248934 (10RobH) [20:05:59] 10DBA, 10Data-Services, 10Quarry: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Mike_Peel) Hmm, Quarry is still not completing the query, and now paws has stopped working. :-( [20:43:50] 10DBA, 10Data-Services, 10Quarry: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Bstorm) labsdb1011 is suffering badly lately, it seems. I see some rough replag and the connection errors spike periodically on that host https://grafana.wikimedia.org/d/00000... [20:45:43] 10DBA, 10Data-Services, 10Quarry: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Bstorm) I can also point out that I just tried to query a table with in wikidatawiki with a simple `select * from limit 1` and it just hung. It was a full view, not a... [20:52:26] 10DBA, 10Data-Services, 10Quarry: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Bstorm) I just found that I got the same result when I did it against the underlying table locally. @Marostegui I think there is a problem with the wikidatawiki database on t...