[07:44:39] are you testing backups on cumin1001 ? [07:44:59] or on dbstore1001 ? [07:45:21] cumin [07:47:16] and you are doing a schema change on my screen session (258587.snapshot_test_T210292)? [07:47:28] or it was just me [07:47:37] connecting to it by mistake [07:49:05] anyway, I will use es2001 and es2002 myself [07:49:32] and cumin 2001 [07:50:06] what? [07:50:23] my screen session is T210713 [07:50:24] T210713: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 [07:50:51] I am not using your screen [07:50:52] I just want to know which host you will not use [07:51:03] so we don't interfere [07:51:11] I am using cumin1001 yep [07:51:14] that is the only one [07:51:28] as you initially said you will test on codfw only, so that confused me [07:51:45] ah for the dumps, yes, for the snapshots I wanted to use 1001 [07:51:46] sorry [07:51:51] ok [08:02:26] may I ask if you are using patch set 17? [08:03:25] for some reason your version is not using a busy loop for waiting, so I am surprised (I did not fix that) [08:12:08] yes, patch 17 [08:12:31] strange, in past executions, I had 1 cpu at 100% [09:02:08] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo) [09:02:13] marostegui: until T218188 is fixed or workaround it, don't run more than 1 backup at a time [09:02:14] T218188: cumin not thread/multiprocess-safe ? - https://phabricator.wikimedia.org/T218188 [09:02:27] sure [09:02:29] thanks [09:15:50] 10DBA: wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError - https://phabricator.wikimedia.org/T218189 (10jcrespo) [09:16:23] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo) [09:16:25] 10DBA: wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError - https://phabricator.wikimedia.org/T218189 (10jcrespo) [09:34:18] hello, just in case one of my favorite dba is around. I have a dummy questions about innodb. I noticed an empty database has some rather huge files created and I am wondering whether they can be tweaked down / disabled ;) [09:34:34] ibdata1 is 76 MBytes (which I guess is going to hold the actual data) [09:34:55] and then I have ib_logfile0 and ib_logfile1 which are each 48 Mbytes, I am guessing they are for transaction logs [09:39:32] to defragment ibdata1, you need to create it from 0 [09:39:59] to reduce log file size you can configure innodb_log_file_size [09:40:04] ah [09:40:17] you may need a restart after that [09:40:45] the context is a dummy mysql database being initialized on CI. It is hold in a tmpfs and I am looking at reducing the default size on creation [09:40:56] I guess innodb_log_file_size is what I was looking for! [09:41:14] jynus: thank you, you are magic as usual! [09:43:26] you can also redunce the extent size [09:43:40] if it is really going to be empty [09:44:40] you can also tweak many things if it is for creation + immediate destroy [09:44:55] like disable the double write buffer [09:44:58] disable binlogs [09:47:18] also looking to separate innodb-undo-tablespaces to aboiv growth of ibdata1, as well as using innodb file per table [09:48:52] if you have code to show I can give it a look [09:51:13] jynus: for now I am just going to raise the tmpfs size. Then yes I guess one day we can schedule a pairing session to finely tweak the settings ;) [09:51:35] I just went with the default :/ [09:59:05] enabling compressin would also be very disk-efficient and would reduce the default growth [10:01:17] I am sure you can also hint at some settings that are unsafe but would be good enough for CI and speed it up a bit [10:01:39] the reason we went to use tmpfs is that it nicely speed up the mediawiki test suite run ;) [10:01:41] yes, I already mentioned some above [10:03:41] maybe even disabling innodb and using only myisam, but some of those extreme changes may actually give CI errors if too different from produciton [10:12:16] jynus: do you mind if I feel a task against my system and #dba and we follow up on phabricator? It is not urgent, just something nicer to have and can be discussed over a few days/weeks ;) [10:12:42] (I prefer phabricator tasks over IRC since that keeps a journal, and mailling lists are hmm a pain) ;D [10:13:23] sure, but make it low [10:16:05] thanks :) [10:30:38] 10DBA, 10Quibble: Optimize MySQL settings for MediaWiki CI / Quibble - https://phabricator.wikimedia.org/T218196 (10hashar) [10:31:57] 10DBA, 10Quibble: Optimize MySQL settings for MediaWiki CI / Quibble - https://phabricator.wikimedia.org/T218196 (10hashar) p:05Triage→03Low Finding some nicer settings for the CI database configuration is a nice to have, that can happens over the course of a few weeks. Hence setting to low priority. [10:36:57] 10DBA, 10Quibble: Optimize MySQL settings for MediaWiki CI / Quibble - https://phabricator.wikimedia.org/T218196 (10jcrespo) One thing that would also speed up the thing, among many other ideas, would be to not run mysql_install_db every time, but having a prepared empty data dir that is copied every time. The... [12:19:42] be careful dbstore1001 is at 87% /srv [12:20:14] that is a lot of space still, but there could be some older stuff that could be deleted [13:41:06] Yeah, there is stuff in archive that we can probably delete [13:41:12] 1.9T snapshot.s1.2019-03-11--15-45-50 [13:49:49] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10jcrespo) {F28380917} {F28380921} [13:50:14] <3<3 [13:51:04] from provisioning servers to zarcillo backup_mariadb.py? [13:51:08] Ah, for the stats [13:51:52] it is all super-clear! [13:52:42] :-D [13:55:56] 10DBA, 10Quibble: Optimize MySQL settings for MediaWiki CI / Quibble - https://phabricator.wikimedia.org/T218196 (10Krinkle) > the huge `l10n_cache` table This can also be put on disk instead, e.g. using php array files (preferred), or cdb. Those also have the benefit of generating faster, which makes the ins... [14:59:57] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10Marostegui) I have been testing the snapshots too with `daily_snapshot.py` What I have seen is also mostly related to what we discussed about th... [15:12:03] I am updating the wiki, but there is like 10 mB of text to update [15:18:02] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10jcrespo) Yeah, `only_postprocess: True` is ignored by `daily_snapshot.py`. I wonder if I should either implement it or error on it? There are s... [15:19:47] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10Marostegui) Yes, not a big deal if it is done now or after the doc (I agree - I would prefer the doc before the check) as it is mostly something... [15:20:54] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10jcrespo) while I agree with the interactive usage of `backup_mariadb.py`, I am not sure about `daily_snapshot.py`, as it is not supposed to be r... [15:25:32] 10DBA, 10Patch-For-Review: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292 (10Marostegui) Agree that it will mostly be run from cron, but not always. ie: snapshot that fails Depending on how often we do snapshots it might... [16:03:57] 10DBA, 10Wikimedia-Site-requests, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10aaron) >>! In T215107#4963154, @Tgr wrote: > So that's an impressive cas... [17:29:02] marostegui: avalaible? [18:26:00] 10DBA, 10Operations, 10ops-eqiad: dbproxy1012 power supply without power - https://phabricator.wikimedia.org/T217394 (10Cmjohnson) 05Open→03Resolved