[06:00:30] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3269817 (10Marostegui) db2041 is done: ``` root@neodymium:~# for i in `cat /home/marostegui/T162611`; do echo $i; mysql -hdb2041.codfw.wmnet --skip-ssl $i -e "show create table revision\G";done bgwiki ************... [06:40:53] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3269857 (10Marostegui) Quite nice!: ``` root@db2071:/srv/sqldata/enwiki# ls -lh ores_classification.ibd... [06:58:25] 10DBA, 06Labs, 10wikitech.wikimedia.org, 07Schema-change: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3269904 (10Marostegui) 05Open>03Resolved a:03Marostegui Tables dropped. This is what has been run: ``` root@silver:~# for i in `mysql --skip-ssl lab... [06:59:44] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3236866 (10Marostegui) a:03Marostegui [07:00:01] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3269911 (10Marostegui) a:03Marostegui [07:09:47] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3269918 (10Marostegui) [07:11:34] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3236866 (10Marostegui) I am currently altering s2 for another maintenance task so I will take t... [07:36:30] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3269935 (10Marostegui) I have done all `nlwiki` on codfw just to make sure all worked as expect... [07:50:41] did you see https://phabricator.wikimedia.org/T161296#3268474 ? [07:51:07] Ah no [07:51:08] I didn't [07:51:12] <3 !!! [07:55:59] it look a bit psycodelic at times: https://grafana-admin.wikimedia.org/dashboard/db/mysql?orgId=1&from=now-6h&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=dbstore1001 [07:57:56] haha yeah, it will be nice to see it on dbstore1002 which will have a "normal" pattern [08:01:29] 1002 is trusty, not supported [08:19:29] I want to reimage db2052 [08:20:42] db2052 or db2062? (as I saw it is depooled) [08:21:06] db2052 [08:21:27] what's up with that one? (just for me to know) [08:21:37] it is trusty [08:21:40] ah [08:21:43] go for it! [08:24:11] I do not know why db2062 is depooled [08:24:26] I know why, it is 10.1 [08:24:38] https://gerrit.wikimedia.org/r/#/c/349388/ [08:24:41] yeah I was checking that [08:24:58] and reading the task [08:25:19] I can pool it back [08:25:47] no rush I would say [08:25:58] I will need to depool another host later, so I can take care of that too if you like [08:27:56] jynus: now that I think of it, the lag in the big value could be the maximum detected [08:28:40] ? [08:30:23] not sure what you mean with that, godog [08:33:08] jynus: the big value displayed at the top, could be topk(1, mysql_slave_status_seconds_behind_master{instance="$server:9104"}) [08:33:28] so it'll do the right thing in both cases, i.e. display the max [08:33:49] anyways, not important [08:35:27] not a fan [08:36:10] not a strong opposition either [08:42:40] yeah I think it is fine like it is now [08:43:25] note that the hosts that have multisource, the max is not that important [08:43:30] *for the hosts [08:44:30] ideally I would have an array there, but the important parameter is heartbeat, not the lag from show slave status [08:44:58] which will be the next thing to try, once we have upgraded all servers to the new exporter [08:52:05] marostegui, you know why I tell you about the servers I reimage, right? [08:52:16] aside from letting you know? [08:52:32] yes, for the maintenance? [08:52:38] yeah [08:52:54] if you do alters though replication, they will end up there, too [08:53:06] but if you do per server stuff, it may be skipped [08:53:29] so we need to be in sync about that [08:53:42] sure sure [08:53:55] so far I am not doing anything that might interfere with your upgrades I believe, am I? [08:53:59] I also ask you just in case [08:54:06] that could happen [08:54:14] sure sure [08:54:17] e.g. you are running a long running schema change [08:54:44] it is funny, you say "sure sure" (I do the same) [08:55:07] No, but I haven't seen anything that could interfere on both directions, so far [08:55:21] but Sean told me when I entered that that sounded obnoxius, so it must be a Spanish thing? [08:55:28] hahaha [08:55:29] could be [08:55:53] What did you use instead of "sure sure"? [08:56:29] no, I meant I wrote things like [08:56:37] "of course of course" [08:56:45] or "yes yes" [08:57:01] and people said bad things about that [08:57:15] oh really? then it must be a spanish thing [08:57:19] because it sounds natural to me [08:57:19] but that is how I speak [08:57:22] me too [08:57:47] in fact, for me is more "humble" [08:58:11] but apparently it is not in English or in Australian [08:58:16] interesting! [08:58:22] never thought about it [08:59:38] http://www.urbandictionary.com/define.php?term=Sure%2C%20sure. [09:00:16] I think the tone is missing [09:00:34] it even has a urbandictionary entry! [09:00:35] wow [09:01:33] Yeah, I guess the tone is missing, but if it is common to read it like that, I guess native speakers assume we are using that tone [09:05:53] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3270117 (10Marostegui) codfw is now done after running it on the master and let it replicate (dbstore200... [10:14:42] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3270383 (10Marostegui) Executed on db1069 (sanitarium) and db1095 (sanitarium2) to replicate downstream... [11:02:53] you should clone https://gerrit.wikimedia.org/r/operations/software/wmfmariadbpy [11:04:21] it is empty right? [11:04:25] for now [11:04:26] I am going for lunch, but checked it out [11:04:28] ok ok :) [11:04:33] just making sure it was correct to be empty [11:04:40] yes [12:14:28] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270793 (10Marostegui) nlwiki done on: sanitarium (db1069) and replicated to labsdb1001 and la... [12:29:36] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270856 (10Marostegui) nlwiki done on all the hosts now: ``` root@neodymium:/home/marostegui/gi... [12:29:49] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270857 (10Marostegui) [12:46:07] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270915 (10Marostegui) Started with `trwiki` directly on codfw master so all codfw gets done (d... [12:53:04] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270921 (10Marostegui) db1069 done and replicated downstream to the labs hosts: ``` root@neodym... [12:58:18] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270930 (10Marostegui) trwiki is done on all the hosts (dbstore2001 will get it tomorrow as it... [12:58:26] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3270931 (10Marostegui) [13:29:04] vote vote vote https://jira.mariadb.org/browse/MDEV-12832 [13:30:00] Another one…:| [13:30:06] ha ha [13:30:20] voted [13:30:37] so I created 10.1.23-1 [13:31:05] note sure if it is worth to create 10.2.23-2 with the upgraded library (not in debian stable) [13:31:28] we would miss mariabackup, which may be interesting to test? [13:31:57] I can upload but not upgrade the ones that had already been done? [13:32:04] we can, but we (you really) are already testing mydumper, so it is not likely we will start testing another solution just yet no? [13:32:14] so maybe not worth wasting your time on another package? [13:32:19] o, of course [13:32:35] just to keep it available, it doesn't affect the server [13:32:50] maybe I can upgrade the client only package [13:35:45] if that doesn't give you headaches, sure [13:38:14] db1041 with 5m of lag [13:39:32] yes [13:39:36] it is the pt-table-checksum [13:39:37] it is depooled [13:39:47] that server gets lagged as soon as it gets some load :( [13:40:14] can't wait to get rid of it [13:41:43] it is ok [13:41:51] it has load 0 to allow that [13:42:14] I just wanted to know if there was heavy querying ongoing [13:42:33] :) [13:42:42] if we consider pt-table-checksum as heavy, then yep! [13:47:16] dump and root filters are most welcome [13:47:19] on activity [13:49:56] Re:mariabackup, think the possiblity of using it as a node-cloning technique for automatic provisioning with galera [13:54:58] I love the idea of having a node to clone others from [13:56:12] well, that is because you have not played yet much with galera yet to hate it :-) [13:56:56] but that is essentially our goal in the end- it just xtrabackup doesn't work for us (but other options will) [13:59:19] hahaha [13:59:33] yeah [13:59:36] but it may be interesting for non-core usages [13:59:44] well, we can certainly try it [14:00:03] (e.g. lists functionality, I mentioned that, but they had no hw) [18:39:22] 10DBA, 10Community-Wikimetrics, 10Icinga, 06Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3272009 (10jcrespo) [18:42:20] 10DBA, 10Community-Wikimetrics, 10Icinga, 06Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3272026 (10Paladox) I vote to migrate to use the mariadb class as it is maintained :) [18:44:37] 10DBA, 10Community-Wikimetrics, 10Icinga, 06Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3272032 (10jcrespo) Paladox- if this is for your own usage- you can do that now, what I cannot guarantee is that it will fullfill your needs easily (for e... [18:45:48] 10DBA, 10Community-Wikimetrics, 10Icinga, 06Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3272033 (10Paladox) Oh, i think the puppet role I'm using right now (deployment_server) is using mysql. Since somehow it is using the mysql class.