[00:44:41] 10DBA, 10Analytics: Wikimetrics is trying to use more than 10 connections at the same time - https://phabricator.wikimedia.org/T173062#3517627 (10Milimetric) [00:44:54] 10DBA, 10Analytics-Kanban: Wikimetrics is trying to use more than 10 connections at the same time - https://phabricator.wikimedia.org/T173062#3517640 (10Milimetric) [00:50:22] 10DBA, 10Analytics-Kanban: Wikimetrics is trying to use more than 10 connections at the same time - https://phabricator.wikimedia.org/T173062#3517671 (10Milimetric) Well, that was easy. https://gerrit.wikimedia.org/r/#/c/371218/ fixed it. Sorry for the noise. [01:38:54] 10DBA, 10Patch-For-Review: duplicate key problems - https://phabricator.wikimedia.org/T151029#3517814 (10Marostegui) s6 had issues with the following tables (on db2046 and db2076): frwiki: linter, user_properties, watchlist, wbc_entity_usage (fixed already on db2046) jawiki: user_properties (fixed already on... [07:00:11] 10DBA, 10MediaWiki-Watchlist, 10Wikimedia-General-or-Unknown, 10Wikimedia-log-errors: User with 40000 entries in their Watchlist cannot access it on Commons anymore: Database error - https://phabricator.wikimedia.org/T171898#3517923 (10jcrespo) Based on logs, this is getting worse on commonswiki, but unlik... [08:27:33] 10DBA: Migrate s5 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3515549 (10jcrespo) While this has to happen somehow, I now think that s4 has more priority, because it is lagging, and dragging other shard (mainly, s1) with it. So we should move probably either s1 or s4 first. [08:40:20] 10DBA: Migrate s5 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3517964 (10jcrespo) [09:57:37] 10DBA: Migrate s5 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518062 (10Marostegui) s4 could fit too. It is taking around 700G on db1095, so that fits in db1102 with no issues (2T available). And that would leave db1095 with around 1T free. [09:59:27] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518067 (10Marostegui) [10:06:14] I have uploaded 10.1.26 and 10.0.32 for jessie [10:06:50] I am also sending a patch to default to systemd on jessie if 10.1 is installed [10:08:14] will upload 10.1.26 for strech later [10:19:24] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518079 (10Marostegui) Not sure how xtrabackup would work (or how InnoDB would behave) given that db1095 has an ibdata1 for all the shards now. So copying it to db1102 and only bringing up s4 might be an issue or could be an issu... [10:20:48] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518083 (10jcrespo) check that mydumper copies the triggers, too. [10:23:09] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518084 (10Marostegui) >>! In T172996#3518083, @jcrespo wrote: > check that mydumper copies the triggers, too. Very good point! Looks like we can do it: ``` --triggers, -G Dump triggers ``` Although before... [11:49:45] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3518131 (10jcrespo) You said: > s5 and s6 will soon be ready to be added too Can you comment which are ready here, because I know you are doing lots of things and I do not want to touch them unless you... [11:52:38] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3518132 (10Marostegui) >>! In T168409#3518131, @jcrespo wrote: > You said: >> s5 and s6 will soon be ready to be added too > > Can you comment which are ready here, because I know you are doing lots of... [11:53:42] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3518133 (10Marostegui) [11:54:12] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3363748 (10Marostegui) [15:28:02] 10DBA, 10Analytics-Kanban, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3518622 (10elukey) From db1046: ``` MariaDB [log]> SELECT count(table_name) FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='log' and not DATA_TYPE='binary' and COL... [15:31:11] 10DBA, 10Analytics-Kanban, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3518626 (10jcrespo) Yes, I assume if not given, it will take the default from the database- changing it on the db everywhere should work for new creations. Do we need t... [15:52:19] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3518637 (10jcrespo) s5 is being loaded right now. I may have configured the memory usage too high for 7/8 instances (15 GB for the buffer pool)- we may want to increase it for dbstore2002, and decrease... [16:02:01] 10DBA, 10Analytics-Kanban, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3518660 (10Marostegui) >>! In T170952#3518622, @elukey wrote: > From db1046: > > ``` > MariaDB [log]> SELECT count(table_name) FROM INFORMATION_SCHEMA.COLUMNS WHERE TA... [16:14:40] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3518678 (10jcrespo) [16:14:42] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3518677 (10jcrespo) [16:16:39] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3515549 (10jcrespo) I am going to drop s6 from sanitarium- it is running out of space (probably because s3), and s6 is the "easier" to reimport if there is any problem. I want to keep db1069 around for longer, in case there are i... [17:09:53] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3518866 (10jcrespo) [17:22:31] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3518887 (10Halfak) 1. Hard to say. N for each page rendering where N is the number of wikidata "usages" on the page.... [17:23:53] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3518901 (10Halfak) One more thing. Part of our test deployment to elwiki as described in this task is to help get a s... [17:58:07] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3519094 (10Halfak) [17:59:15] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#2825398 (10Halfak) [17:59:34] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#2825398 (10Halfak) [20:20:15] 10DBA: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3519520 (10Marostegui) >>! In T172996#3518677, @jcrespo wrote: > I am going to drop s6 from sanitarium- it is running out of space (probably because s3), and s6 is the "easier" to reimport if there is any problem. I want to keep... [20:29:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3519559 (10Marostegui) MariaDB changelog (https://mariadb.com/kb/en/mariadb/mariadb-10032-changelog/) says it has been indeed fixed on 10.0.32: ``` Revision...