[08:59:02] morning folks, I'm still looking for code reviews for the 3 changes needed to get the new thanos-be nodes into service; I'd like to start that process first thing Monday, so reviews today would be kind? [08:59:20] https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/8 to teach the ring manager about codfw rack D3's network [08:59:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1093884 to add the new nodes to swift::backends [08:59:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1093885 to actually add them to the rings [09:33:30] {done} [09:33:51] TY <3 [09:34:46] arnaudb: can you look at the gitlab MR too, please? [09:37:01] I ticked the "reviewed" box but maybe I'm missing a button to push let me check [09:41:51] don't see anything more I could do Emperor, am I missing a button somewhere? [09:42:40] I think clicking "approve" is usual [09:42:57] (or the thumbs-up) [09:43:07] Emperor: o/ I think that the DCops folks didn't give an explicit green light but I'll resolve the racking tasks, you are free to go from my pov [09:43:37] I can't unfold the approvers list, I think this might be coming from ACLs? anyway, I've added a 👍 :) [09:43:57] thanks :) [09:44:04] ^-- applies to both arnaudb and elukey :-D [09:56:49] after hammering the commonswiki api 40 times because db query timeouts, they finally bootstrapped [10:02:51] jynus: ? [10:04:03] the media backups, I meant [10:12:27] ah :) [10:12:54] elukey: are you and jhathaway happy with the SM Config-Js now, or would you like to keep one (or more!) of the ms-be* nodes for further testing / debugging? [10:14:12] Emperor: we'd need one/two nodes for testing if possible, the issue with the BMC is still open, Supermicro is responsive but we'll need to do some tests for usre [10:14:16] *sure [10:14:17] would it be a problem? [10:25:08] elukey: absolutely not, have as many as you like :) [10:25:48] (but I'll start putting the others into production next week) Would you like 1 per DC (2), 2 per DC (4), some other number? [10:29:02] Emperor: need to check with Jesse, I'd say a couple for DC are enough.. keep in mind that the hosts need another reimage etc.., and I haven't touched eqiad yet [10:29:10] so there is some provisioning work to do [10:31:58] I'm happy to leave them all alone for now if that's going to result in less confusion :) [11:22:16] finally! https://phabricator.wikimedia.org/T376892#10347738 [11:31:10] 👍 [11:34:29] Emperor: do you have an estimation of how much space you will have usable for ms-be after the expansion? would 1 Petabyte be enough? [11:35:02] to backup originals? [11:36:56] those are two quite different questions :) [11:38:35] I know [11:39:02] I don't need exact numbers, more like an approximation [11:40:54] commons originals are currently 574 TB ( https://commons.wikimedia.org/wiki/Special:MediaStatistics ) [11:42:38] nah, that's not right [11:42:58] I have 656 178 463 691 108 bytes in backups [11:43:08] it doesn't have into account deleted or older versions [11:43:18] and commons is like 99% of content [11:44:19] In eqiad, we're replacing 9 old nodes (ms-be105[1-9]) with 9 new nodes. That's effectively replacing 9x12x4T with 9x24x8T, so an increase of 1296T. But we 3x replicate everything, so that's actually a capacity increase of about 400T [11:44:51] ok, so that matches the 1 PT more or less [11:44:54] I have [11:44:57] now [11:45:50] About 2/3 of our capacity is used for originals (the other 1/3 is thumbs) [11:49:32] We're currently using 3320558095597568 / 4866186333814784 of ms capacity in eqiad (that's about 3.3 / 4.9 PB) [11:50:04] have I approximately answered your questions sufficiently? [11:50:24] Our actual limit is the size of the thumb databases, which is a whole other can o'worms [11:50:25] yes, more than sufficient [11:52:18] this validates my expansion, at least until you expand again [11:52:29] thanks [11:52:51] NP [11:53:01] * Emperor just made their first edit to commons [11:53:29] (I was annoyed the commons stats page didn't link to the mediastatistics page) [11:53:58] yeah, but those are not that useful for us, as those are user stats (latest version, etc.) [11:54:10] useful for editors, but not as much for backend maintainers [11:56:47] Mmm, not sure if there's a nicer way of answering your question than "swift stat" on the 256 originals containers & the 256 deleted-objects containers... [11:57:14] no, no worries, I know that because I gather my own stats [11:57:36] it was the expansion what I didn't knew about, and you already answered me [12:02:45] 👍 [12:05:01] I checked and commons is a 99.3% of original size [12:05:19] *originals