[01:30:06] <wikibugs_>	 10DBA, 10MediaWiki-Categories, 03Community-Tech-Sprint, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3051978 (10Bawolff) I guess theres always the question of what happens when someone makes an even longer collation name.   (Consider me...
[07:16:18] <wikibugs_>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3052329 (10Marostegui) db2069 ``` | revision | CREATE TABLE `revision` (   `rev_id` int(8) unsigned NOT NULL AUTO_INCREMENT,   `rev_page` int(8) unsigned NOT NULL DEFAU...
[09:18:17] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3052487 (10Marostegui) >>! In T154485#3050716, @Marostegui wrote: > db1036 and db2035 were still lagging behind more than 30k: >  > ``` > ˜/jynus 19:26> it is now doi...
[09:20:34] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3052496 (10jcrespo) I wouldn't be against it, but I would ask to start puppetizing that- Something like a replication filter, and add those through hiera with a new p...
[09:27:56] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3052512 (10Marostegui) >>! In T154485#3052496, @jcrespo wrote: > I wouldn't be against it, but I would ask to start puppetizing that- Something like a replication fil...
[09:55:54] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3052564 (10Marostegui) I have taken a look at all the hosts in s1 (enwiki) and the table there already has the PK, so...
[09:58:44] <jynus>	 BTW, I did the increase of threads for the pool with sql.py
[10:00:24] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3052574 (10TTO) Note that if the PK already exists, the unique index may not exist, so it would just be a matter of ad...
[10:02:49] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3052576 (10Marostegui) >>! In T155605#3052574, @TTO wrote: > Note that if the PK already exists, the unique index may...
[10:08:17] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3052579 (10TTO) Eh? Where are we dropping the `ug_group` index? From https://gerrit.wikimedia.org/r/#/c/328377/22/main...
[10:19:36] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3052589 (10Marostegui) >>! In T155605#3052579, @TTO wrote: > Eh? Where are we dropping the `ug_group` index? From http...
[11:16:40] <jynus>	 labsdb1005 may have crashed yesterday?
[11:17:32] <marostegui>	 looks so
[11:19:02] <marostegui>	 doesn't look like a oom
[11:19:29] <marostegui>	 We think the query pointer is invalid, but we will try to print it anyway.
[11:19:32] <marostegui>	 Query: OPTIMIZE TABLE `ipp_changes`
[11:19:34] <marostegui>	 which doesn't mean it was that one though
[11:20:44] <jynus>	 it complainms about several tables, but those are probably myisam
[11:20:58] <jynus>	 nothing we can do if people want to create tables on /dev/null
[11:22:38] <jynus>	 but that is why it is almost impossible to get a consistent replication to the slave
[11:23:15] <marostegui>	 yeah, it has been complaining for years, but as you said, not much we can do
[11:23:24] <marostegui>	 s/years/days obviously 
[11:23:30] <marostegui>	 I am going for lunch, will be back later
[11:23:43] <jynus>	 see you
[14:08:37] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - https://phabricator.wikimedia.org/T70942#3053003 (10Marostegui) 05Open>03Resolved I have, by chance, arrived to this old ticket, which I believe can be closed.  labsdb1001 runs 10.0.15...
[14:24:24] <marostegui>	 is it worrying that I know most of the wikis in which shard they are from the top of my head?
[14:26:38] <jynus>	 no
[14:26:55] <jynus>	 you were surprised I knew that a few months ago
[14:27:15] <jynus>	 you achieved enlightment now, too
[14:27:31] <marostegui>	 hahaha
[14:27:34] <jynus>	 but wait a few months, I will change all of them
[14:28:29] <marostegui>	 Make sure you clean up s3, please
[14:28:54] <jynus>	 clean up?
[14:29:11] <marostegui>	 yes, if you are going to change all of them, leave s3 with less wikis
[14:29:14] <marostegui>	 spread them
[14:29:16] <marostegui>	 :p
[14:29:17] <jynus>	 yes
[14:29:31] <jynus>	 specially some of the largest
[14:39:37] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3053102 (10Marostegui) Apart from s1:  s2 all databases there have the PK on their user_groups table s4, commonswiki h...
[14:41:22] <jynus>	 ^did you use sql.py ?
[14:47:42] <marostegui>	 I did! -h multi :)
[14:49:14] <marostegui>	 I had to look at the code though to see if there was an option for ALL the wikis, so maybe having —shard SHARD or even providing a list of available shards ie: available options,s1,s2,m3,m4 whatever can be useful
[14:49:22] <marostegui>	 or if you prefer just —shard-all as an option
[14:49:29] <marostegui>	 because I tried: "all"
[14:49:36] <marostegui>	 and didn't think it could be capital letters
[14:49:57] <jynus>	 I think it is --shard ALL
[14:50:04] <marostegui>	 it is
[14:50:23] <jynus>	 I will have to redo that part
[14:50:27] <jynus>	 on my own usage
[14:50:37] <jynus>	 I realized that there are several scenarios
[14:50:45] <jynus>	 all servers
[14:50:52] <jynus>	 all servers, one per wiki
[14:50:55] <marostegui>	 can you also pass a list of hosts?
[14:51:21] <jynus>	 one per table
[14:51:24] <jynus>	 etc.
[14:51:32] <jynus>	 only masters
[14:51:35] <jynus>	 etc.
[14:51:57] <jynus>	 I can separate the list generation from the exec
[14:52:08] <jynus>	 (it is already separate on code)
[14:54:33] <marostegui>	 I would also say that when using multi, it might be good to ignore temporary hosts down
[14:54:36] <marostegui>	 for example
[14:54:47] <marostegui>	 root@neodymium:/home/jynus# ./sql.py -h multi -e "show create table user_groups" --shard ALL --no-dry-run
[14:54:50] <marostegui>	 ERROR: Could not connect to labsdb1001.eqiad.wmnet:3306/arwiki
[14:54:53] <marostegui>	 and it stops there
[14:55:22] <marostegui>	 maybe it would be good t have something like —ignore-errors or something similar
[14:57:18] <jynus>	 it could be an option
[14:57:25] <jynus>	 I wanted it to stop by default
[14:57:38] <jynus>	 if it was an alter you would want to do some action, then continue
[14:58:15] <marostegui>	 that is true, but maybe for selects or stuff like that, having an option to ignore hosts down or slow
[14:58:24] <marostegui>	 I wonder why i cannot connect to labs1001 btw
[14:58:26] <marostegui>	 lets see
[14:58:35] <jynus>	 it is the ssl
[14:58:44] <jynus>	 needs an upgrade + reboot
[14:59:46] <marostegui>	 ah :)
[15:20:53] <jynus>	 I think that is a bug that may actually be fixed on the repo version
[15:21:01] <jynus>	 let me check
[15:40:22] <jynus>	 I think it should work now
[15:40:33] <jynus>	 ERROR: Could not connect to dbstore1001.eqiad.wmnet:3306/frwiki
[15:40:38] <jynus>	 but then it continues
[15:42:02] <jynus>	 don't use it yet for updates or alters, ok?
[15:42:21] <marostegui>	 no, of course not
[15:43:24] <jynus>	 I would add a format option
[15:43:35] <jynus>	 one that is pure text
[16:22:05] <wikibugs_>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1070 - https://phabricator.wikimedia.org/T158969#3053253 (10Marostegui)
[16:23:07] <wikibugs_>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1070 - https://phabricator.wikimedia.org/T158969#3053239 (10Marostegui) Hey @Cmjohnson it should be safe to change this disk when you have time Thanks!
[16:33:51] <wikibugs_>	 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 06Stewards-and-global-tools (Temporary-UserRights): Schema changes for expiring user groups - https://phabricator.wikimedia.org/T155605#3053297 (10Marostegui) And the list of wikis of s3 that DO NOT HAVE the PK but an UNIQUE key instead (this is mostly f...
[17:20:15] <marostegui>	 ETA 45 minutes to finish the transfer to dbstore1001!
[17:41:21] <jynus>	 :-)
[17:48:28] <wikibugs_>	 10DBA, 10Wiki-Loves-Monuments-Database: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517#3053448 (10Lokal_Profil)
[17:50:59] <marostegui>	 15 minutes left, my plan would be to start —skip-slave-start do the mysql_upgrade, restart…and cross my fingers?
[17:57:30] <jynus>	 mysql_upgrade will take 30 minutes
[17:57:33] <jynus>	 maybe more
[17:57:53] <jynus>	 also the scheduler will start and will break replication stuff
[17:58:22] <jynus>	 there is a start option to disable the event_Scheduler
[17:58:26] <marostegui>	 lets see
[17:58:35] <jynus>	 not "break"
[17:58:42] <jynus>	 but hope you understand
[17:58:50] <marostegui>	 yes yes, i do :)
[17:59:05] <jynus>	 we need to get rid of the event
[17:59:07] <marostegui>	 --event-scheduler=DISABLED
[17:59:25] <marostegui>	 Starting the MySQL server with the https://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_skip-grant-tables option causes https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_event_scheduler to be set to DISABLED,
[17:59:26] <jynus>	 a service would be much more easy to handle
[17:59:39] <jynus>	 I think you may have to restart after that
[18:00:03] <jynus>	 in any case, check twice all repl threads
[18:00:38] <marostegui>	 we can: —skip-slave-start —event-scheduler=disabled then run mysql_upgrade, then restart with —event-scheduler=disabled again, then check all the threads, start replication
[18:00:45] <marostegui>	 and when we are good, we can enable the scheduler
[18:01:04] <marostegui>	 i would need to leave around 8-8:15 by the way
[18:01:17] <marostegui>	 4 minutes left!
[18:02:05] <jynus>	 you shouldn't have stayed late
[18:02:26] <marostegui>	 neither have you and here we are :p
[18:02:29] <jynus>	 there is really no need
[18:02:36] <jynus>	 yeah, but I was here to do other things
[18:03:25] <marostegui>	 i want to leave it catching up the weekend, and I got to advance a bit on that schema change ticket, and got to see a few slides from fosdem finally! 
[18:04:38] <marostegui>	 ok, so /etc/init.d/mysql start —skip-slave-start —event-schduler=disabled sounds good to you?
[18:05:03] <marostegui>	 scheduler obviously
[18:06:32] <jynus>	 it should be
[18:06:57] <jynus>	 but I have almost giving up trying to have a sane dbstore :-)
[18:07:03] <marostegui>	 haha
[18:07:07] <marostegui>	 ok, so the transfer is done
[18:07:10] <marostegui>	 let's go ahead
[18:07:22] <marostegui>	 you want to tail the error log to see the fun?
[18:07:32] <jynus>	 sure
[18:08:16] <marostegui>	 here we go!
[18:09:12] <jynus>	 check if toku is active
[18:09:17] <jynus>	 before everthing
[18:09:20] <marostegui>	 ok
[18:09:22] <jynus>	 if you hve the change
[18:09:25] <marostegui>	 cannot log in yet
[18:09:26] <jynus>	 like reading a table
[18:09:35] <jynus>	 oh, true
[18:09:39] <jynus>	 it is take some time
[18:09:42] <marostegui>	 yeah
[18:09:50] <jynus>	 like 2001
[18:09:53] <jynus>	 now
[18:09:55] <marostegui>	 yeop
[18:10:07] <jynus>	 (but much faster)
[18:10:18] <marostegui>	 tokudb plugin seems loaded as per show plugins at least
[18:10:26] <jynus>	 strange
[18:10:31] <jynus>	 that it works
[18:11:10] <jynus>	 I do not remember fixing the systmctl flags
[18:11:40] <marostegui>	 and i can read a table with tokudb
[18:11:52] <marostegui>	 arwiki.abuse_filter_action for instance
[18:11:56] <jynus>	 that is cool
[18:12:02] <jynus>	 I believe you
[18:12:03] <marostegui>	 let's go for the mysql ugprade?
[18:12:08] <jynus>	 yes
[18:12:15] <jynus>	 I just do not know when that was fixed
[18:12:37] <jynus>	 Unsupported sql_command=46
[18:12:41] <marostegui>	 yep
[18:12:54] <marostegui>	 Repairing tables
[18:12:54] <marostegui>	 mysql.master_status
[18:12:54] <marostegui>	 Error    : CONNECT Unsupported command
[18:12:54] <marostegui>	 Error    : Can't lock file (errno: 122 "Internal (unspecified) error in handler")
[18:12:57] <marostegui>	 error    : Corrupt
[18:13:04] <jynus>	 really?
[18:13:17] <marostegui>	 it is related to mysql.master_status and slave_status tables only
[18:13:18] <marostegui>	 (so far)
[18:13:22] <jynus>	 ah
[18:13:24] <jynus>	 good
[18:13:30] <marostegui>	 yeah, so far so good
[18:13:40] <jynus>	 I thought it was regarding data tables
[18:13:44] <marostegui>	 no no
[18:13:45] <marostegui>	 just mysql
[18:13:48] <jynus>	 I couldn't care more about those
[18:13:53] <jynus>	 *less
[18:13:53] <marostegui>	 XD
[18:14:05] <marostegui>	 now on phase 2, fixing views
[18:14:06] <jynus>	 In theory, this upgrade should be easy
[18:14:13] <jynus>	 that is what it takes 30 minutes
[18:14:17] <marostegui>	 only tokudb worries me
[18:14:26] <jynus>	 opening every single files and db
[18:14:34] <jynus>	 you can go take a coffe here
[18:15:53] <jynus>	 I've just sent you some literature for your amusement
[18:16:04] <marostegui>	 hahaha
[18:16:06] <marostegui>	 lets see
[18:16:55] <marostegui>	 wtf!!
[18:17:25] <marostegui>	 haha they mention your nickname
[18:17:53] <jynus>	 date: 2007
[18:17:57] <marostegui>	 I was going to ask
[18:17:58] <marostegui>	 haha
[18:18:12] <jynus>	 23 servers on Seoul
[18:18:14] <marostegui>	 what about those other that they mention, still active?
[18:20:10] <jynus>	 surprisingly, most of them do
[18:20:40] <marostegui>	 polish was really popular at that time I can see!
[18:20:47] <marostegui>	 It says, polish right?
[18:21:04] <jynus>	 it is hard to read
[18:21:18] <jynus>	 I may get a better version
[18:21:34] <jynus>	 yes
[18:21:42] <jynus>	 it still is, remember pl and pt
[18:21:48] <jynus>	 on which shard?
[18:22:18] <jynus>	 you learn a lot about iso codes for you resume!
[18:22:58] <marostegui>	 it is weird because internet in poland I am sure it wasn't that developed at the time
[18:23:23] <jynus>	 from eswiki, I think the excuse was "botting"
[18:23:32] <jynus>	 not sure the reality
[18:29:23] <jynus>	 do you know what would fix most of our pains with many tables?
[18:29:33] <marostegui>	 I have replied to you with something for your amusement too
[18:29:34] <jynus>	 the new MySQL data dictionary!
[18:29:46] <marostegui>	 that one you tested and reported a bug for? :)
[18:29:50] <jynus>	 yeah
[18:29:56] <jynus>	 but it is really fast
[18:29:57] <marostegui>	 was it fixed?
[18:30:05] <jynus>	 yes, it was
[18:30:16] <jynus>	 but no more milestone releases 
[18:30:26] <jynus>	 unless it has gone unnoticed
[18:30:45] <marostegui>	 you've got the bug report somewhere?
[18:30:49] <jynus>	 lol
[18:30:54] <marostegui>	 hahaha
[18:30:57] <marostegui>	 I guess you so the email
[18:30:58] <marostegui>	 XDDD
[18:31:04] <marostegui>	 *Saw
[18:31:25] <jynus>	 year?
[18:31:29] <marostegui>	 2006
[18:31:36] <marostegui>	 XDD
[18:32:14] <jynus>	 https://bugs.mysql.com/bug.php?id=83706
[18:32:29] <jynus>	 it was a stupid bug, actually
[18:32:44] <jynus>	 lack of quotes in binary because it is a keyword
[18:33:45] <marostegui>	 haha i see the last answer yeah
[18:33:50] <jynus>	 yea, not 8.0.1 yet
[18:34:09] <jynus>	 but I saw the data dictionary in action
[18:34:22] <jynus>	 and it may sound stupid when they say
[18:34:37] <jynus>	 10x more performance for information_schema
[18:34:43] <jynus>	 who cares?
[18:34:59] <jynus>	 but then here you are, running mysql_upgrade for half an hour
[18:35:07] <jynus>	 mostly doing show tables :-)
[18:35:07] <marostegui>	 hahaha that is a good point
[18:35:29] <marostegui>	 although it is a bit crazy to have 900 databases in once place too :p
[18:35:40] <jynus>	 oh, I do not disagree
[18:35:55] <jynus>	 however
[18:36:00] <jynus>	 I've seen it worse
[18:36:07] <marostegui>	 yes me too
[18:36:08] <marostegui>	 and i cried
[18:36:09] <jynus>	 100000 databases
[18:36:18] <jynus>	 it no longer worked
[18:36:18] <marostegui>	 100k??????
[18:36:20] <jynus>	 yep
[18:36:24] <jynus>	 don't ask
[18:36:25] <marostegui>	 I saw 60k and I cried
[18:36:35] <marostegui>	 100k? does that even start?
[18:36:39] <jynus>	 maybe they had smaller tables
[18:36:52] <jynus>	 I think they were installations of 
[18:37:09] <jynus>	 what is the name of the open source course stuff?
[18:37:54] <jynus>	 moodle
[18:38:06] <jynus>	 now imagine supporting a whole country, all public system
[18:38:11] <jynus>	 schools
[18:38:16] <jynus>	 in a single mysql instance :-)
[18:38:28] <marostegui>	 oh moodle
[18:38:29] <jynus>	 one db per class and year
[18:38:37] <marostegui>	 :o
[18:38:55] <marostegui>	 did mysql even start with 100k databases???
[18:39:09] <jynus>	 it did
[18:39:18] <jynus>	 xtrabackup, of course, it crashed
[18:40:19] <jynus>	 do you know what else would be great?
[18:40:31] <marostegui>	 not having toku? :)
[18:40:32] <jynus>	 multiple-table tablespaces
[18:40:45] <jynus>	 single file per db
[18:41:05] <jynus>	 5.7 has it
[18:41:25] <jynus>	 you will not solve the table problem
[18:41:30] <jynus>	 but at least the file problem
[18:41:32] <marostegui>	 yes, I was going to say that it sounded familiar that they were going towards that direction
[18:43:01] <marostegui>	 we are now on mrwikibooks already
[18:43:07] <jynus>	 pfff
[18:43:27] <jynus>	 at this point I will tell you to go away :-)
[18:43:34] <marostegui>	 Yeah, I think I will
[18:43:38] <marostegui>	 you can do screen -x transfer
[18:43:40] <marostegui>	 on dbstore1001
[18:43:45] <marostegui>	 and it is there
[18:43:55] <jynus>	 events disabled, right?
[18:43:59] <marostegui>	 yep
[18:44:00] <jynus>	 so just reboot?
[18:44:14] <marostegui>	 events disabled with —event-scheduler=disabled
[18:44:23] <marostegui>	 so if you restart without it they will come back
[18:44:27] <jynus>	 yeah, I was going to reboot with no parameeters
[18:44:32] <marostegui>	 ah, then yes
[18:44:33] <jynus>	 and let things happen
[18:44:42] <jynus>	 when it is finished, of course
[18:44:49] <marostegui>	 nywiki
[18:44:51] <marostegui>	 already
[18:45:09] <jynus>	 see you next week
[18:45:39] <marostegui>	 pfl
[18:45:42] <marostegui>	 I might stay!!!
[18:45:47] <marostegui>	 it is doing phabricator
[18:46:02] <jynus>	 you know wikidata and zhwiki and plwiki and ptwiki...
[18:46:37] <marostegui>	 ptwiki is already done, but wikidata, yes pfff
[18:52:45] <marostegui>	 I need to leave the house now :_(
[18:52:59] <marostegui>	 Email me to let me know if all went good, so I can have a relaxing dinner!
[18:53:20] <jynus>	 ok, bye!
[18:53:29] <marostegui>	 it is doing urwiki now
[18:53:32] <marostegui>	 almost there
[18:53:35] <marostegui>	 see you later!
[18:53:39] <marostegui>	 thanks for all the help
[18:56:40] <jynus>	 restarting now
[22:52:56] <wikibugs_>	 10DBA, 10MediaWiki-Categories, 03Community-Tech-Sprint, 13Patch-For-Review: Increase size of categorylinks.cl_collation column - https://phabricator.wikimedia.org/T158724#3054267 (10kaldari) >I guess theres always the question of what happens when someone makes an even longer collation name. If anyone make...