[21:01:21] * robla gets ready to start E263 [21:01:39] \o/ [21:02:12] hi [21:02:39] #startmeeting ArchCom: Schema change for page content language (T69223) [21:02:39] Meeting started Wed Aug 24 21:02:39 2016 UTC and is due to finish in 60 minutes. The chair is robla. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:02:39] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:02:39] The meeting name has been set to 'archcom__schema_change_for_page_content_language__t69223_' [21:02:39] T69223: Schema change for page content language - https://phabricator.wikimedia.org/T69223 [21:02:54] #topic Wikimedia meeting channel | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ [21:03:08] hi [21:03:11] hi everyone! [21:04:17] jynus wanted to make sure we have developer consensus on T69223 . do we? [21:04:17] T69223: Schema change for page content language - https://phabricator.wikimedia.org/T69223 [21:05:07] the main issue seems to be with people thinking we could apply schema changes optionally [21:05:22] and that was everybody's mind when it was merged [21:05:48] Would it be problematic to apply the schema change to all wikis? [21:05:51] To me the question is whether it makes sense semantically. is the language really a property of the page? or is it rather a property of the content? can it change over time? cane (with Muti-Content-Revisions) have content in multiple languages on the same page? [21:05:53] then, 2 years later, people realize that we may not need that extra column on one of our largest core tables [21:06:04] ^legoktm [21:06:48] that is the point, people think we can do that, and even if we could technically do that, that would mean wmf forking mediawiki code and maintaining its own version [21:07:17] er, I don't think we need to go down that rabbit hole... [21:07:18] legoktm: on the ticket jynus seems to be pushing back on that pretty strongly [21:07:20] Heya! :) [21:07:23] DanielK_WMDE_: Yes. Very much so. [21:07:25] no [21:07:29] I do not want that [21:07:41] so i'm a little vague on the scope here. are we talking about applying a schema change to enable an existing feature, or talking about how best to do that kind of feature? [21:07:49] James_F: yes what? it's a property of the content, not the page? [21:07:54] brion: the former AIUI [21:08:12] I want people merging changes to have into account how practical that is [21:08:16] TimStarling: from what I read, it seemed to me that he was only objecting to the plan of not applying it to all wikis. [21:08:20] if that is the case, it should go into the new content table we discussed last week, not into the page table [21:08:25] #info topic discussed: do we believe the page_lang field is needed on all wikis? [21:08:34] DanielK_WMDE_: well, this feature has been in core since 1.24 [21:08:36] DanielK_WMDE_: Yes, sorry. And also yes, we might in future have parallel slots with different languages. [21:08:39] it just hasn't been deployed yet [21:08:47] most of the people that agreed that change: aaron, nemo, etc. [21:08:59] thought we were going to apply it only to a few tables [21:09:16] if we are not going to have that, do we still want it (that way) [21:09:44] James_F: if that is the case, i vote for adding this to the content table, and not deploy it to the page table on more wikis. we'd just have to remove it again in the future. [21:09:58] some people mentioned that maybe the slot thingie we discused last week could be an alternative [21:10:05] I do not know, what I want [21:10:12] is people agreeing [21:10:18] that would also be an opportunity to apply the same optimization we want to apply for storing content model & friends. use int ids to represent the language in large tables [21:10:24] and not asking for the reverse change in a few weeks [21:10:26] my inclination is to agree with DanielK_WMDE_; it feels like it belongs with the content and we're already planning to change that with the content direction [21:10:55] let me give my personal take a bit offtopic [21:10:55] so if there's not a pressing need to enable this with the existing page_lang schema tweak, we save ourselves some trouble by making only one change later [21:11:04] :) [21:11:10] merging code is "easy" [21:11:19] it is not, but physically easy [21:11:23] I wonder how it would work with indexing - different languages require different analyzers, etc. but right now all pages of the wiki are in the same index... [21:11:43] aplying schema changes in a runing system is not that easy, and technically cannot be reverted [21:11:56] so I want the maximum amount of agreement possible [21:12:02] *nod* [21:12:04] not to the point of blocking things [21:12:07] to avoid this from happening in the future, I think needs to explicitly state that if a schema change is proposed for a table, it must be applied to all wikis that have that table. [21:12:11] it is ok to make mistakes [21:12:24] but making sure that here and now we all agree [21:12:34] that is my only requuest to all of you [21:12:37] :-) [21:12:40] SMalyshev: the content handler could declare a different index depending on the content language... but it would have to declare an index of all the languages we plan to support, right? would that be an issue? [21:13:05] hooking up the search index to multiple languages is a question for later, but a good one :) [21:13:10] jynus: ++ [21:13:14] is Nemo around? [21:13:20] (In - https://www.mediawiki.org/wiki/Extension:Translate/Usability_improvements_2014#Tentative_timeline - what does "University starts-Work hours shifted to evening and night" refer to please? As some of you know, CC MIT OCW-centric in 7 languages and Yale OYC-centric CC was donated to Wikidata last autumn 2015 by WUaS ... and per Lydia:) too) [21:13:21] DanielK_WMDE_: I don't know... it probably needs to be discussed with ebernhardson and dcausse [21:13:23] jynus: i agree - not all changes are easy to undo. anything that creates artifacts (data) needs extra thought. [21:13:35] the other thing [21:13:37] #info jynus: [schema changes on Wikimedia's production cluster are tricky] so I want the maximum amount of agreement possible [21:13:44] is that that was merged a long time ago [21:13:51] I wasn't part of the wmf [21:14:00] so maybe needs shifted, etc. [21:14:13] brion: like everything, there's no actual urgency, but this is a bug request from 2007 that a volunteer fixed 2 years ago and was released as part of 1.24 and in active use by other non-Wikimedia wikis. [21:14:40] that is another issue [21:14:47] legoktm: right, so any future replacement would need to properly migrate from both states [21:14:53] I think more important than the particular thing [21:15:02] DanielK_WMDE_: letting the content handler choose the index doesn't sound too great, more likely search would have to key off the language it reports at and decide what needs to happen [21:15:04] there is a disconnection between mediawiki and wmf [21:15:05] #info we might in future have parallel slots with different languages. it feels like [the language] belongs with the content and we're already planning to change that with the content direction [21:15:06] so to block code written 2 years ago on the basis that it will conflict with code that doesn't exist yet seems pretty silly? poor to me. [21:15:11] which is not bad by itself [21:15:25] but I would like the coordination to be smoother [21:15:48] #info legoktm: [no actual urgency, but T69223] is a bug request from 2007 that a volunteer fixed 2 years ago and was released as part of 1.24 and in active use by other non-Wikimedia wikis. [21:15:48] T69223: Schema change for page content language - https://phabricator.wikimedia.org/T69223 [21:15:52] for this I only have requests, not real solutions [21:15:56] it's the deployment that's hard part :) [21:16:08] robla: the actual 2007 bug is https://phabricator.wikimedia.org/T11360 [21:16:10] legoktm: I agree that the code should not have been merged without a plan to deploy it. But possibly you're not saying that? [21:16:22] is there something we can do to make that gap smaller? [21:16:31] #info 14:16:08 robla: the actual 2007 bug is https://phabricator.wikimedia.org/T11360 [21:16:44] but lets focus on this bug first [21:16:59] jynus: we could have a MediaWiki core maintenance team that coordinates with ops/db .... but that's another question for another time [21:17:04] ha ha [21:17:08] fair enough [21:17:23] jynus: mediawiki governance currently lies with the wmf... people are discussing an independant mediawiki org of sorts. do you think that would make things better, or worse? [21:17:41] both [21:17:47] lol [21:17:48] brion: Except at the time of 1.24 we /had/ such a team… [21:17:51] To have a multilingual page doesn't especially conflict with the possibility to assign one language to a page. Different extensions/needs could receive different solutions. So there is no obvious conflict between content language and page language. [21:17:55] if I am being a bit selfish, (please undertand what I mean) [21:17:55] ori: hehe... [21:18:11] DanielK_WMDE_: if you're trolling, please do it elsewhere. come on. [21:18:15] I do not care about mediawiki, I care about users using mediawiki to serve wikis :-) [21:18:52] James_F: There was always a plan to deploy it, it's just that that plan was slightly flawed in the case where they weren't planning to deploy to all dbs. But now that it has been identified, we can fix it by deploying the schema change to all wikis [21:18:55] anyway, this is easy to go offtopic [21:19:09] For example, the page language could be set at null for a monolingual wiki or a wiki with pages with multicontent language-specific (or a mul code there). [21:19:29] I think we will not be able to take a decision about this in 20 minutes, but can someone think about it more deply, offline? [21:19:51] I don't think we need to discuss page_lang design, that was done 2 years ago [21:19:56] should we try to deploy this to a few wikis to sync with mediawiki [21:19:58] James_F: at the time of 1.24 we didn't have a dba and were basically unwilling to change schemas of large tables in prod though :P [21:19:59] legoktm: it was a genuine question. sometimes, formalizing relationships makes communication easier. but as robla said, it's a question for another time. [21:20:04] jynus: as a practical matter, do we have an estimate on how long it would take to deploy page_lang column as written on enwiki page db? [21:20:09] hours/days/months? [21:20:15] there are two open questions: what to do about the rest of the schema changes, and policy going forwards [21:20:16] Aaron said it's 40MB to add the column to enwiki (https://phabricator.wikimedia.org/T69223#2285536), and AFAIS no one else has identified performance problems with the schema change. [21:20:33] brion, times does not matter much [21:20:37] #info 14:20:16 there are two open questions: what to do about the rest of the schema changes, and policy going forwards [21:20:45] So I don't see any reason to further block this schema change [21:20:47] cause if it's a quick change then our lives may be simplified by saying 'apply it' so we have a consistent set of dbs [21:20:59] as I think (cannot guarantee) that I can do it online as it has a PK [21:21:00] then can concentrate on being saner in future :D [21:21:19] my vote: don't deploy page_lang to more wikis, add cont_lang when we add the content table. [21:21:22] but I will require read only at the start and end of the application [21:21:28] maybe multiple times [21:21:30] yeah, I think I would like the schema change to be scheduled for the next maintenance batch [21:21:37] So if I've well understand the fear of jynus, it was: "is someone going to revert this page_lang field in the next weeks/months?". If so, jynus advices to defer the deployment, if we're sure this change is stable enough, jaime is okay to deploy it to all wikis at the next opportunity. I've well summarized your thoughts jynus ? [21:22:05] doing all the master switches is a lot of work, but each schema change applied during that maintenance period is not a lot of extra work [21:22:30] info: I have already a bunch of changes scheduled: watchlist, others [21:22:41] oooh, watchlist... [21:22:41] and I asked for a week of maintenance [21:22:43] you need read-only time during the master switches, right? [21:23:04] yes, the idea is sync it with the dc failovers [21:23:07] #info applying changes to page table probably not super slow (should be onlineable) and a bunch of other changes are queued up for next master switch [21:23:11] apply it to the passive dc [21:23:39] jynus: is Dereckson's summary accurate? [21:23:55] brion, it is actually "super slow", just that for most of the time it can live at the same time than regular writes [21:24:12] and definitely exclusive from other alters on the same table or even server [21:24:16] #info it is actually "super slow", just that for most of the time it can live at the same time than regular writes [21:24:39] relatively serial due to the extra writes involved [21:25:15] I tend to apply schema changes serialy for extra safeness, not needed if a dc is passive, though [21:25:50] so tes, although with tables like page or revision, I would say "in the next year" [21:25:56] jynus, TimStarling: if(!) we end up putting the language into the content table, and we deploy that change in half a year, page_lang would become redundant. if it was certain that we are going to do that, would you want to deploy page_lang now, for consistency? [21:26:24] DanielK_WMDE_: I'm pretty sure that we should deploy it now for consistency, yes [21:26:33] possible cases: page_lang becomes a summary field, or gets removed [21:26:48] TimStarling: mit take is yes if it's cheap, but not if it's painful... [21:27:05] if in 6 months or a year there's a patch to remove the field, it can be scheduled in the same way [21:27:11] how dependent core code is from that field, once the feature is enabled? [21:27:13] so i feel like we're asking jynus if it'll be painful, and jynus is saying pain is ok if it's necessary :) [21:27:25] hehe [21:27:26] jynus will be angry but we will promise to try to do better next time [21:27:28] jynus: should be only a couple bits pulling from it [21:27:36] yes, I just want you to make that decision [21:27:37] brion: yea, it's kind of circular :D [21:27:40] and very easy to switch back off if needed [21:27:48] it felt that that was not a real decision before [21:28:10] "I know that undoing this will be painful and I will not complain if jaime delay it" [21:28:10] I don't really know if we will have cont_lang or whatever, that has not been the subject of an RFC meeting as far as I know [21:28:22] so yeah i'm ok with saying 'deploy it, make sure everything's consistent' even if it ends up changing later [21:28:42] i suspect we'll obsolete it but need to be more thoughtful about doing so [21:29:06] TimStarling: well, the original concern was "let's not do it if we are going to undo it in a few months". it's a real possibility. it should be considered. [21:29:11] it's also conceivable we'll store content language through different metadata [21:29:17] as I said, I am ok with someone looking at it in depth and sending a report for everybody to discuss offline [21:29:23] i'm not opposed to deploying page_lang now. i'm trying to find out if it is "necessary" [21:29:35] I just want more eyes on it [21:29:45] brion: can be a page_prop even [21:29:46] [14:29:06] TimStarling: well, the original concern was "let's not do it if we are going to undo it in a few months". it's a real possibility. it should be considered. <-- that was not the original concern. [21:30:44] DanielK_WMDE_: it can't be a page_prop, https://phabricator.wikimedia.org/T69223#2501293 [21:30:49] but yeah overall we need to not do stuff like this in future. ;) either deploy everywhere or nowhere, it gets confusing to have optional schema tweaks [21:30:59] ++++++++1000000 [21:31:14] #info we may wish to promulgate an RfC that just says "thou shalt not have optional schema updates" [21:31:22] in releng/ops we do not have support for such a schema [21:31:24] like legoktm said, that can be in the policy document on mw.org [21:31:25] thanks brion :-) [21:31:27] legoktm: then i must have misunderstood what jynus said. [21:31:42] in fact, I mention that every difference from mediawiki core is a bug [21:31:57] we don't need a separate RFC, that is a DBA decision and jynus is making it [21:31:58] and that has already caused issues [21:32:31] TimStarling: I think it makes sense for it to have some sort of policy backing somewhere [21:32:33] +1 [21:32:41] I was bold and did it: https://wikitech.wikimedia.org/w/index.php?title=Schema_changes&action=historysubmit&type=revision&diff=818276&oldid=542880 [21:32:42] well, it's a coding decision to make the update optional, then a dba decision to have deployed it or not [21:32:52] legoktm: yay [21:33:09] legoktm: thanks! [21:33:26] the question was, "people seemed to want this optional, that may have been an option some time ago (I do not know), it is not longer an option because it is causing multiple problems" [21:33:31] brion: well, we tend to write code for optional schema changes since typically the code is deployed before the schema change, so it needs to be able to work without it. [21:33:52] we have in the past had optional schema updates for minor things like varchar widths [21:33:56] jynus: I'm not sure why people seemed to want this optional, perhaps it was only a suggestion to ease your dba life not to apply it where not needed [21:33:56] DanielK_WMDE_: we may wish to do that the other way round :) [21:34:04] #info: legoktm add "If a table is deployed to multiple wikis/databases, any schema changes must be applied to all wikis where that table is present for consistency." to [[wikitech:Schema_changes]] [21:34:10] there is no serious arguments against coherence [21:34:23] The person I have been discussing this more was pushing for optionally applying it [21:34:33] brion: yeesss.... but then the code needs to be able to live with uninitialized fields.... [21:34:49] but that was in a different environment, where schema updates were harder and inconsistency was less of a problem [21:34:59] legoktm: we should update https://www.mediawiki.org/wiki/Development_policy#Database_patches too [21:35:41] in fact, those schema differences are #1 cause of slowing down schema changes [21:35:45] heh "Make your schema change optional – All schema changes must go through a period of being optional. Some examples:" [21:35:48] * brion headdesk [21:35:59] that is ok, in contecty [21:36:05] it means code optional [21:36:11] those aren't quite wrong yeah :D [21:36:14] and it is still good advice [21:36:30] you cannot expect a schema change to be applied atomically [21:36:35] yea, that's what i meant. [21:36:40] so code must be able to ignore it [21:36:41] that is ok [21:36:56] and a good policy, just needs to be clarified [21:37:18] "period =/= all the time" [21:38:39] so, where are we with this particular change? [21:38:43] so, thing I would want from you, can someone compromise to give a look at the code and say "yes, consensus to apply to all wikis" [21:39:23] and then, compromise for involvement on deployment plans, with my help in future cases [21:39:44] if you need a new policy [21:40:25] it would be "schema change need to include a plan for deployment, or something" [21:40:26] https://www.mediawiki.org/w/index.php?title=Development_policy&type=revision&diff=2223295&oldid=2142163 [21:40:34] robla: ^ [21:40:38] TimStarling: do you think that someone should step forward and do what jynus is asking for? [21:41:28] I'm not really clear on what jynus is asking for [21:42:03] there is 3 ways to go here: [21:42:15] I apply this change to all wikis [21:42:26] i believe jynus needs go/no go for this schema change, but also for us to have a clearer policy in future on making sure schema changes are consistent and someone works with ops on future schema-changing merges [21:42:34] I apply this change to a subset of wikis, as requests (which I do object highly) [21:42:40] I do not apply this change [21:43:05] what do we need to do to make such a policy real? enough to edit the pages or do we need to approve an RfC or organize ourselves with the other devs? [21:43:08] whatever it is, a reasonable set of reasons, not matter if finally it is not the most optimal decision [21:43:23] I've been pretty emphatic that I think the first should be done, apply to all wikis [21:43:35] i'm ok with that [21:43:44] but without code review, there won't be consensus on whether page_lang is a good idea, from a blank slate design point of view [21:43:56] good or not it's already there ;) [21:44:28] so we're in agreement to deploy it to all wikis? [21:44:30] #info jynus: a) I apply this change to all wikis, b) I apply this change to a subset of wikis, as requests (which I do object highly), c) I do not apply this change [21:45:15] so as i understand tim says a), i'm leaning to a), DanielK_WMDE_ thought maybe c) ? is that current? [21:45:35] or are we still fluid :) [21:45:47] :) [21:45:49] but nobody things b) is awesome [21:45:52] *thinks [21:46:12] note that a decision now is not binding forever, I just want to unblock on this topic [21:46:26] i'm good with a) if nobody complains if page_lang becomes redundant in half a year. i'm not certain that it will, but it might. [21:46:53] option b) doesn't sound good, it doesn't solve any of the underlying propblems [21:47:00] *nod* [21:47:10] the second question is, I suppose this was an undesired state (?), is there something I/we can do to avoid it in the future (short term actionables)? [21:47:15] *nod* too [21:47:16] right, I'm saying a over c just as a judgement call, based on expected lead time before an alternative solution becomes available, cost in terms of DB space, cost in terms of schema change time [21:47:56] i'll trust TimStarling that he's judging that right, then [21:48:27] right :) [21:48:31] should we put "a)" in last call? [21:48:35] jynus: in terms of process? we should make sure we have a db/ops person at these RFC meetings, i think. [21:48:39] er..."final comment"? [21:48:51] jynus: I think correct, we don't want to be left in this state in future. not sure exactly solution but we agree on the problem statement there :D [21:48:55] DanielK_WMDE_: I don't think we need to insist on that [21:49:19] I think from a process perspective, we just need to make sure https://www.mediawiki.org/wiki/Development_policy is right [21:49:34] robla, that is outdated [21:49:47] I worked with developers for a better process [21:50:01] jynus: it's a wiki; please edit (like legoktm just did) [21:50:10] I already did [21:50:18] robla: not insist, of course. i think it would help. but it doesn't have to be the live meeting. comments on rfcs in pahb will also do. we should make sure to reach out to db/ops whenever an rfc entails a schema change or other ops issue [21:50:20] it would be cool if we had more than one DBA so that jynus didn't have to reliably be here at stupid o'clock [21:50:22] but as a non-"mediawikiean" [21:50:25] i think we have become better at this already. [21:50:31] I edited wikitech [21:50:32] do we need to do more in this regard? [21:50:37] let me share [21:50:43] TimStarling: indeed [21:50:52] robla, https://wikitech.wikimedia.org/wiki/Schema_changes [21:51:22] that is my requested process to speed up schema change applications, work with me from earlier, use some tags when ready for deployment [21:52:09] also an explanation of why I am so slow applying schema changes [21:52:14] :-) [21:52:23] (spoiler: I am not) [21:52:38] jynus: are you saying that anything we put on mediawiki.org with respect to database stuff is irrelevant? [21:52:45] no [21:52:55] but I have to for schema-changes [21:53:00] ok i have to run, taking cat to vet for a checkup :) /meow [21:53:01] *had to fork [21:53:17] because it was full of discussions and abandoned proposals [21:53:36] and create blocked-on-schema-change for the deployment part [21:53:48] WMF servers != mediawiki process [21:54:34] you can do anything you want they way you want, but on deployment, I am the process quality filter, if you are ok with that [21:54:42] we deploy MediaWiki code weekly, so WMF servers is nearly equal to mediawiki process [21:55:13] yes, but releng wants to know anything about deploying schema changes [21:55:17] *nothing [21:55:27] so I had to create a process myself [21:55:54] could you link to your process from https://www.mediawiki.org/wiki/Development_policy ? [21:56:02] yes, I can add that [21:56:11] thanks! [21:56:20] I just do not want to impose anything to developers [21:56:32] I worked on that in fact woth several developer to be more agile [21:56:47] and agreed to that when deploying schema changes [21:57:29] cool, I think we're in a good place here now [21:58:18] Great, all! [21:58:22] so, outcome of today: 1. last call on option A) above and 2. clarifying policy [21:58:27] sound about right? [21:58:46] (er....*1. final comment on option A) [21:59:18] yes [21:59:25] :) [21:59:30] so I will clarify the "optional part" add outdated tags and link to the policy [22:00:01] wonderful, thanks jynus! will #endmeeting in 60 seconds [22:00:03] I will send a link to the changes to the ticket in case you think I've become too evil [22:00:18] oh hey domas! you just missed a fun discussion about the deployment of schema changes ;) [22:00:30] haha [22:00:35] schema change? what is that?! [22:00:43] *g* [22:00:43] that thing you have automatized [22:00:52] jynus: I don't think any of us thinks you're evil :-) [22:00:53] ah domas [22:00:59] hi Tim! [22:01:10] but people on the wiki cannot agree on :-) [22:01:11] we could have had a much more fun discussion if you showed up an hour ago :) [22:01:24] alright, ending meeting.... we can have after meeting chat [22:01:28] was busy not looking at my screen! [22:01:31] #endmeeting [22:01:32] Meeting ended Wed Aug 24 22:01:31 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [22:01:32] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-08-24-21.02.html [22:01:32] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-08-24-21.02.txt [22:01:32] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-08-24-21.02.wiki [22:01:32] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-08-24-21.02.log.html [22:01:38] for example, we discovered schema changes applied on the db, and not on code [22:01:48] and some applied only to enwiki [22:01:56] and some applied only to some servers [22:02:02] that happens! [22:02:08] were they sitting there for past 10 years? [22:02:34] some of them I teorize had been done by tim based on the dates [22:03:02] domas had to have a hand in some of them as well, then [22:03:06] enwiki.recentchanges and enwiki.watchlist has/had special indexes, I think. [22:03:09] so now I am planning of having a database of schema changes [22:03:28] we should make a chart of who was DBA at each point in history [22:03:29] on a related note... why are many tables that are VARCHAR in tables.sql VARBINARY on the live system? [22:03:31] yes, we maintain that on logging and revisions, but at least is documented [22:03:44] they are not varbinary [22:03:49] DanielK_WMDE_: they're varbinary because charset is binary [22:03:58] they are varchar that with binary colation [22:04:00] varchar in binary is varbinary [22:04:06] they are esentionally binary strings [22:04:18] domas: in the declaration, they are varchar binary - which the docs say is varcahr with binary collation, not charset... [22:04:21] because we started with MySQL 4.0 and it didn't have varbinary [22:04:38] TimStarling: I think 4.0 had varbinary just no collations [22:04:50] and we should continue having them as varchars [22:05:02] for the people that use utf8 or utf8mb4 [22:05:04] IIRC it had a varbinary keyword for forwards compatibility but it was implemented as an alias to varchar binary [22:05:04] jynus: as far as i can see, they are actually varbinary - and thus have the wrong length in many cases. [22:05:12] yeah, could be [22:05:34] I do not think wrong lenght is the right workd [22:05:39] we were storing utf8 in latin1, and then I did proper varbinary conversion on 5.1 upgrades [22:05:41] short lenght is a better one [22:05:54] it was a good move [22:06:12] we had one CHAR() field that caused me some grief [22:06:15] even now, I do not think mysql supports the latest unicode standard [22:06:19] but most of that transition was not detected by anyone [22:06:34] at some point we should make the comment of revisions larger [22:06:35] jynus: it was a good move in a way how every major transition that is not detected by anyone ;-) [22:06:56] jynus: term_text in wb_terms on wikidatawiki, for instance. it's causing https://phabricator.wikimedia.org/T142691 [22:06:56] jynus: facebook patch has a fix where covering indexing doesn't get broken by prefix [22:07:04] oh, well. [22:07:09] domas: I manually upgraded 90% of our servers from 5.5 to 10 [22:07:18] jynus: There are/were a lot of debates about rev_comment and whether it should be a blob or a pointer to a blob or. [22:07:18] so don't tell me [22:07:19] jynus: that is not an upgrade [22:07:28] jynus: upgrade would be 5.6 ;-p [22:07:43] well, it uses a more updated innodb [22:07:46] that is better [22:08:19] also, in 5 days a new DBA will arrive [22:08:27] 7 days [22:08:30] Working with you? [22:08:35] I hope [22:08:37] :-) [22:08:39] yay [22:08:46] maybe against me :-) [22:09:12] at that point we could get some actual work done [22:09:15] in the super-early days, DBA duties were shared between a few volunteers: JeLuF, James Day, myself [22:09:15] so sad that you force people to work on maria [22:09:23] jynus' greeting: "welcome! (I think)" :-) [22:09:40] i miss JeLuF! [22:09:55] I did a period of being essentially sole DBA later on, maybe between domas and Asher? [22:10:06] mark was a DBA too [22:10:15] he once fixed full filesystem when I was on a boat [22:10:38] problem is features arrive and arrive [22:10:44] that was when I automated master switches [22:10:48] my peers still laugh how I had to fix wikipedia from some restaurant in central oregon [22:10:54] haha, good job [22:11:08] I did a master switch with under a minute of read-only time [22:11:22] we can do better now with GTID [22:11:31] this was with many shards [22:11:35] fully automatic [22:11:57] good stuff [22:11:58] got to go [22:12:06] attaboy Tim [22:12:37] gotta sleep [22:12:44] I thought I did sub-minute switchovers [22:12:46] domas: you weren't in Prineville "fixing Wikipedia", were you? [22:12:46] good, productive takj [22:12:48] wasn't a fancy script [22:12:54] jynus: thanks for staying up! [22:12:57] robla: Bend [22:13:20] though I probably visited Prineville on that trip [22:13:51] * robla has driven up and down US 97 way more times than he really wanted to [22:15:10] I've never been to operational facebook datacenter [22:15:18] I only went to one when they were pouring concrete [22:15:45] last operational datacenter I've been to was pmtpa ;-) [22:16:34]