[07:07:21] 10DBA, 06Operations: defragment db1015, db1035 and db1027 - https://phabricator.wikimedia.org/T110504#1579303 (10Marostegui) db1035 looks good now ``` root@db1035:/srv/sqldata# df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 1.6T 1.1T 585G 64% /srv ``` [07:10:21] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2955288 (10Marostegui) It looks good now ``` root@db1038:/srv/sqldata# df -hT /srv Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 1.6T 1.1T 572G 65% /srv ``` [07:10:30] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2955289 (10Marostegui) 05Open>03Resolved [07:30:51] 10DBA, 06Operations, 10Wikimedia-General-or-Unknown: Spurious completely empty `image` table row on commonswiki - https://phabricator.wikimedia.org/T155769#2953992 (10Marostegui) Hello, That is indeed present on all the servers for commonswiki. It is also present at enwiki for instance. I have checked oth... [08:42:33] 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#2955384 (10Marostegui) I have started altering db2047 to see how long it takes. [08:45:08] 10DBA: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#2955390 (10Marostegui) [09:34:07] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2955527 (10Marostegui) >>! In T130128#2951866, @Marostegui wrote: > After this change dbstore2001 is catching up quite nicely :-) > So once: https://gerrit.wikimedia.org/r/#/c/332228/ is +1'ed those flags wil... [09:38:55] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2955535 (10jcrespo) >> This is what have changed: >> >> ``` >> default-storage-engine = InnoDB >> #plugin-load = ha_tokudb >> #tokudb_cache_size = 24G >> #tokudb_fs_reserve_percent = 1 >> ``` > > After this... [09:41:00] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2955538 (10Marostegui) >>! In T130128#2955535, @jcrespo wrote: >>> This is what have changed: >>> >>> ``` >>> default-storage-engine = InnoDB >>> #plugin-load = ha_tokudb >>> #tokudb_cache_size = 24G >>> #to... [09:42:46] 10DBA, 06Labs, 10Tool-Labs: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2955540 (10Marostegui) @yuvipanda ok to close this ticket? [09:56:20] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2955552 (10Marostegui) >>! In T145885#2953741, @demon wrote: >>>! In T145885#2951958, @Marostegui wrote: >> That will convert ALL... [10:08:02] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2955567 (10Marostegui) See: T145885#2951958 This change requires us to change 2 global flags on the host. As well as doing the AL... [10:22:42] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 10Revision-Scoring-As-A-Service-Backlog, 15User-Ladsgroup: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2955621 (10Ladsgroup) I wish I would have seen this sooner. A patch is coming, Since i... [10:26:24] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 10Revision-Scoring-As-A-Service-Backlog, and 2 others: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2955626 (10Ladsgroup) p:05Triage>03Unbreak! [10:26:53] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 10Revision-Scoring-As-A-Service-Backlog, and 2 others: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2955631 (10Ladsgroup) I'll add a regression test soon. [12:35:21] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 06Revision-Scoring-As-A-Service, and 3 others: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2945272 (10Ladsgroup) 05Open>03Resolved [13:16:22] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2955937 (10Paladox) I have never tested with a master and a replicate so not sure if it will work. I've only tested it on the main... [13:22:14] 10DBA, 06Labs, 10Tool-Labs: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2955943 (10yuvipanda) Yup! [13:22:36] 10DBA: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#2955944 (10Marostegui) It took around 5 hours to ALTER db2047: ``` root@neodymium:/home/marostegui/git# mysql --skip-ssl metawiki -e "show create table pagelinks\G" --skip-ssl -hdb2047.codf... [13:23:45] 10DBA, 06Labs, 10Tool-Labs: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2955946 (10Marostegui) 05Open>03Resolved [13:48:46] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2955998 (10TTO) p:05Triage>03High [13:49:43] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2956001 (10TTO) There are three options here: either the broken rows are manually removed from the DB, we make TitleValue's error... [13:52:55] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2933244 (10Marostegui) I wouldn't touch the DB manually, but try to fix it in some other way. Who knows what inconsistencies we c... [13:56:40] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2933244 (10jcrespo) The rows can be deleted, but this really needs mediawiki middleware (maintenance script) because those titles... [13:57:46] ^we added a comment at the same time with the same thoughts [13:57:58] yeah [13:58:02] that is good I think! :) [14:10:22] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 06Revision-Scoring-As-A-Service, and 4 others: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2956030 (10Marostegui) Thanks @Ladsgroup! [14:12:59] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2956031 (10TTO) I can promise you those old categorylinks entries with spaces in them are not needed. The category links will hav... [14:41:56] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956039 (10Dzahn) >>! In T155764#2955567, @Marostegui wrote: > This is m2 and the other databases on this host are: > bugzilla_test... [15:15:39] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2956059 (10daniel) TitleValue should generally throw an exception when it gets invalid invalid data. That's what exceptions are f... [15:17:06] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: "Invalid DB key" errors on various special pages - https://phabricator.wikimedia.org/T155091#2956062 (10daniel) Uh.... why are there negative values in cat_pages anyway? That should never happen. Perhaps it's related to th... [15:42:09] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956092 (10Paladox) @demon would you think what @Marostegui said would work. (master to replication)? [17:15:07] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956267 (10demon) >>! In T155764#2955567, @Marostegui wrote: > Given the size of the tables (very small ones) I would rather do the... [18:26:05] 10DBA, 06Operations, 10Wikimedia-General-or-Unknown: Spurious completely empty `image` table row on commonswiki - https://phabricator.wikimedia.org/T155769#2956444 (10matmarex) >>! In T155769#2955296, @Marostegui wrote: > Just to try to understand the whole picture, what is the impact of having that empty ro... [19:18:17] 10DBA, 06Operations, 10Wikimedia-General-or-Unknown: Spurious completely empty `image` table row on commonswiki - https://phabricator.wikimedia.org/T155769#2953992 (10Legoktm) Let's just delete it? Seems similar to T96233.