[01:37:56] 10DBA, 10SRE, 10ops-eqiad, 10Patch-For-Review: Investigate and repool db1134 - https://phabricator.wikimedia.org/T274472 (10Jclark-ctr) @jcrespo Is host still down can maintenance be preformed on host? [04:57:59] 10DBA, 10SRE, 10ops-eqiad, 10Patch-For-Review: Investigate and repool db1134 - https://phabricator.wikimedia.org/T274472 (10Marostegui) Yes, you can proceed anytime Thanks [06:25:44] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1075.eqiad.wmnet - https://phabricator.wikimedia.org/T274235 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db1075.eqiad.wmnet` - db1075.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [06:30:00] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1075.eqiad.wmnet - https://phabricator.wikimedia.org/T274235 (10Marostegui) This is ready for DC-Ops [06:30:11] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1075.eqiad.wmnet - https://phabricator.wikimedia.org/T274235 (10Marostegui) a:05Marostegui→03Cmjohnson [06:31:06] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [07:42:44] elukey: all the dbstores are done? if so, could you update T273281? [07:53:03] marostegui: they are yes, I updated the parent I believe, lemme update also that task [07:53:52] done :) [07:54:26] elukey: grazie! [09:10:09] 10DBA: Migrate codfw sanitarium hosts (db2094/db2095) to Buster and 10.4 - https://phabricator.wikimedia.org/T275112 (10Marostegui) [09:10:53] 10DBA: Migrate codfw sanitarium hosts (db2094/db2095) to Buster and 10.4 - https://phabricator.wikimedia.org/T275112 (10Marostegui) 05Open→03Stalled p:05Triage→03Medium [09:16:06] mariadb 10.5 is what will get distributed with bullseye [09:16:39] not 10.4? [09:19:00] https://packages.debian.org/bullseye/mariadb-server [09:19:15] sweet [09:19:25] I have better news [09:19:29] "From MariaDB 10.6.0, tables that are of the COMPRESSED row format are read-only by default. This is the first step towards removing write support and deprecating the feature." [09:19:30] Good that I mentioned that we need to start testing 10.5 in the annual planning \o/ [09:19:37] WTF [09:20:02] When is bullseye due out [09:20:19] RhinosF1: Don't know, but we haven't even migrated to buster entirely yet [09:20:19] "soon" [09:20:41] marostegui: true [09:21:01] * RhinosF1 has never had to go through updating Debian versions [09:21:44] That compression deprecation is going to be "fun" [09:21:48] And not only for us [09:22:02] marostegui, to be fair, they are not deprecating compression, they are just pushing towards their method: https://mariadb.com/kb/en/innodb-page-compression/ [09:22:45] which uses row format dynamic & PAGE_COMPRESSED=1 [09:23:16] maybe because it causes crashes? [09:23:41] could be [09:23:51] https://mariadb.com/kb/en/storage-engine-independent-column-compression/#comparison-with-innodb-page-compression [10:34:13] I am about to restart dbprov hosts, it will take me probably around 30m-1h, do you plan to do any recovery backup in that time, marostegui, kormat ? [10:35:16] fine for me [10:36:46] I mean, you cannot planned of an unplanned data loss :-), but I will be restarting them serially, stop if I see any problem [10:36:53] *plan [10:36:55] * for [10:43:14] nope from me [10:43:19] you can go ahead [11:35:58] finished with dbprovs, you can use them as usual [11:40:39] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1090.eqiad.wmnet - https://phabricator.wikimedia.org/T274333 (10Marostegui) [12:43:56] jynus: can you double check if the hosts you've rebooted lately are being marked here as well? https://phabricator.wikimedia.org/T273281 [13:40:17] I will when I finish a batch [13:40:39] I have not even marked them on my own ticket :-) [13:40:55] sure [14:05:29] I like to do them in batches so I don't spam summary editing :-) [15:37:31] jynus: of the various *Execution classes in wmfmariadbpy.RemoteExecution, how many are in working condition? i see unit tests only for CuminExecution [15:37:43] @ meeting [15:37:48] ack [15:52:14] depends on what you mean with "working condition" :-) [15:52:57] that has gone through so many hands (started by a volunteer, then me, then Gsoc intern, then you) that I may not remember [15:53:31] In terms of "have you ever run them", I think all worked at some point [15:53:40] and I think the local execution I used it for testing [15:53:57] aside from cumin for production, the others never got a lot of usage [15:54:20] Note that RemoteExecution predates cumin (it was done before it) [15:54:46] if you intend to deprecate others that are not cumin, I think nothing depends on it [15:55:04] i had a quick look at wmfmariadbpy/transferpy/wmfbackups, and yeah, only CuminExecution is used [15:55:15] but I would like to keep the abstract interface unless there is bugs [15:55:40] so we don't depend directly on cumin when it gets changed [15:56:14] keep it, doesn't mean keep it unchanged [15:56:30] it api can and should change if needed [15:56:52] cumin is way more ambitious on what it does than RemoteExecution [15:57:28] but when transfer.py got started, there was no cumin [15:58:55] right, i get that [15:59:39] the other, minor, reason was to be able to port wmfbackups to other non-wmf infra [16:00:41] tell me what is in your mind, I think as long as we can continue running backups, no suggestion will have issues [16:00:47] :-) [16:00:53] or [16:01:42] I can move remoteexecution to wmfbackups repo (e.g. if you want to use cumin directly for wmfmariadb), whatever you prefer [16:02:26] in the short term, i'm thinking of nuking the non-cumin remoteexecution stuff. it's a bunch of unused/untested code [16:02:37] i haven't figured out a longer term plan yet :) [16:02:38] that seems fine to me [16:02:51] I thought you wanted to do super heavy api changes [16:02:53] :-D [16:02:54] jynus: and don't worry, i won't break something you use without talking to you first :) [16:03:08] I thought about doing it myself [16:03:16] but I was like meh [16:03:26] "at some point in the future" [16:03:31] so you will have my +1 [16:03:41] if for some reason the code will be reused [16:03:57] (which is unlikely, it will likely be reimplemented properly) [16:04:06] we can take it from git history [16:04:09] yeah exactly [16:04:23] it is too rotten at this time [16:04:49] yeah, more than the rest, even! [16:05:03] :) [16:05:18] btw, semi related [16:06:20] I think we, you and riccardo commented on integration with the sre automation repo [16:06:44] more integration == better, but it is not clear to me their release policy [16:07:07] I think you worked on that before and it was difficult [16:07:24] "sre automation repo" == spicerack? or something else? [16:07:35] yeah, I get lost with spicerack and the other thing [16:07:43] what's is called [16:08:02] cookbooks maybe? [16:08:04] yes [16:08:13] I don't know much about how it is structured [16:08:25] ah ok [16:08:34] yeah, i've done a small bit with it, [16:08:46] and integrating wmfmariadbpy's libs into it is a definite goal [16:09:07] yeah, specially because there will probably functions that we could use [16:09:17] generic-sre ones, not specific to dbs [16:09:18] yep [16:09:40] but not sure if we would get blocked if we needed an update or something or how releases work [16:09:58] as in, I think it is a goal to do, but not sure how complicated it would be [16:10:18] I am asking not only about wmfmariadbpy, but about wmfbackups [16:10:24] there's always the strategy of annoy volans until he makes a release. :) [16:10:28] and maybe you had a look at it before [16:10:57] and maybe relevant because somethign so generic as RemoteExecution should be there (?) lots of question marks [16:11:06] idk [16:11:12] I'm always annoyed, that's why it's so easy to get a release ;) [quote™] [16:11:13] I need to have a look at it [16:12:19] volans: https://i.imgflip.com/4yj51h.jpg [16:12:37] exactly that one! ;) [16:13:06] I preferred this version :-))))) https://i.imgflip.com/4yj56e.jpg [16:13:12] ;-) [16:13:37] lol [16:14:26] so out of jokes, what is a good thumb rule to integrate directly, vs. maybe just call? [16:14:52] is "we are going to do a lot of realeases because we are building it" a good excuse not to? [16:16:01] the basic rule of thumb for me is: [16:16:06] on one side, maintaining a package and building/upload every times is some overhead [16:16:34] but on the other, it give you more freedom to release for non very mature software, like everything I write [16:16:37] - anything that is big enough and complex enough to leave by itself, do so and just add a wrapper in spicerack that exposes it at the higher level needed in cookbooks [16:17:08] "just add a wrapper" yeah, that is exactly what I meant with just call it [16:17:09] - anything that is smaller enough or not important enough to live on its own put it directly into spicearck [16:17:11] as a workaround [16:17:47] if it's something so generic that doesn't even require root but specific to wmf then wmflib should be the right place