[08:05:20] 10DBA, 06Community-Tech-Tool-Labs, 10Striker, 13Patch-For-Review: Create production database and users for Striker - https://phabricator.wikimedia.org/T142545#2566885 (10jcrespo) ``` root@californium:~$ mysql -h m5-master.eqiad.wmnet -u striker -p -e "SHOW CREATE DATABASE striker" Enter password: +-------... [11:15:42] 10DBA, 06Community-Tech-Tool-Labs, 10Striker, 13Patch-For-Review: Create production database and users for Striker - https://phabricator.wikimedia.org/T142545#2567108 (10jcrespo) ``` MariaDB MISC m5 localhost (none) > SHOW GLOBAL VARIABLES like 'innodb_file_format'; +--------------------+-----------+ | Var... [15:52:31] jynus: that puppet patch I put you on as a reviewer is in a completely different puppet repo from ops/puppet. Having now seen your ops/puppet patch I'm fairly certain I'm at least on the right track for mw-vagrant, but will make it a ::mysql::something class that is optional rather than default. [15:52:57] switching mw-vagrant to mariadb is a whole other task that I should look into at some point [15:53:12] (maybe when we switch to using a jessie base image) [15:53:26] interesting [15:53:45] why does not use the upstream one? [15:53:59] even if it is a different realm [15:54:05] I am not asking you to do it now [15:54:07] we don't have a nice way to reuse things from ops/puppet [15:54:08] I am wondering [15:54:19] but labs do reuse it [15:54:44] allow me to do a +1 [15:54:49] sorry, a +0 [15:55:02] yes, but mw-vagrant is local vms for developers; none of us have +2 in ops/puppet [15:55:23] and feel free to so as you want; sorry, I saw /puppet and assumed it was the upwards one [15:55:24] and being blocked for weeks to get review is no fun :) [15:55:33] yeah, no worries [15:55:52] well, I attended you in, what, 2 days? [15:55:54] I should have left a note explaining why I put you on it [15:56:07] so weeks would be unaccurate [15:56:08] oh yes, you have been very accomidating [15:56:34] in any case, I will deploy the production one [15:56:43] and you will have access [15:56:54] :) thank you [15:57:21] not only labs labs; labs-beta uses it, too [15:57:35] anyway [15:58:17] *nod* beta cluster is just another Labs project; although one that tends to have many local cherry-picks [16:04:35] 10DBA, 06Community-Tech-Tool-Labs, 10Striker, 13Patch-For-Review: Create production database and users for Striker - https://phabricator.wikimedia.org/T142545#2567909 (10jcrespo) This is resolved to me, only keeping it open waiting for confirmation from @bd808 everything worked. [16:28:43] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2567942 (10AlexMonk-WMF) [16:48:58] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2567996 (10AlexMonk-WMF) [17:47:46] jynus: I'm curious, how does wsrep_causal_reads hold up at maintaining read-your-writes under network partitions? [17:48:34] looking at slides of what it does, I get how it helps in the case of no partitions [17:48:54] first, galera or a cluster using it does not have sharding [17:49:14] I mean network partitions [17:49:17] not shards [17:49:18] so all nodes have all content [17:49:26] yes, I am getting there [17:49:44] * AaronSchulz is familiar with certification-based replication in general [17:49:54] so, knowing that, nodes just have a heartbeat [17:50:31] if less than 1/2 of the nodes cannot see the other, they stop accepting connections [17:50:45] however, if you use galera just as replication [17:51:11] you can force a node to ignore split brains [17:52:27] I didn't go into HA considerations for your answer [17:52:42] because I didn't know what the requirements were [17:53:17] but galera does nothing revolutionary, you just need nodes in 3 datacenters to provide a proper service [17:53:50] not necessarily node, there is a galera arbitrator that can make the 3rd node [17:54:40] so you only have 2 data nodes and one extra light service to handle split brains [17:54:47] *can have [17:55:16] is there a window of inconsistency within the heartbeat timeout or is that accounted for somehow? [17:55:37] * AaronSchulz is always skeptical about stuff after reading aphyr ;) [17:55:39] there is a hard and a soft timeouts, all configurable [17:55:48] oh [17:55:57] yes, galera is not serializable [17:56:12] it only works in READ-COMMITED mode [17:56:20] so you can have phantom reads [17:56:46] but that is exactly our current level of isolation for our independent nodes anyway [17:57:36] but please, do not think galera for anything too production-y, not because it is bad, but because you cannot just "put it there" [17:58:07] the only place I would be thinking about it (aside from very special places) [17:58:19] is as an active-passive master, for easier maintenance [17:58:41] but we could do that with an L7 proxy that understands SQL [17:59:16] there is not really much work in that direction currently, I would like us to "fix" regular replication first [17:59:53] and with fix replication, I mean getting rid of undeterministic statements by mediawiki [18:02:24] speak of the devil: https://gerrit.wikimedia.org/r/#/c/305627/ [18:03:25] yes, in theory ids are sent [18:03:32] as part of the transaction [18:04:01] but I believe under less controlled situations, like labs, those are the main casues of inconsistencies [18:04:18] we need to get rid of the model of INSERT...SELECT and archive-like tables [18:04:40] both image and archive are scheduled to be droped at some point [18:04:51] *imagearchive, I think it is called [18:05:16] and for some reason, I was not the only one interested on getting rid of that pattern [18:05:40] in an ideal world, most transactions should be as idempotent as possible [18:06:43] here I am overusing idempotent as "double application should work and o nothing or fail" [18:07:48] thank you for working on that [18:08:09] I will retire for this week, It is getting late here. Have a nice weekend! [18:09:37] see you