[00:51:43] Amir1, hnowlan: I didn't try to figure out if anything is configured to use it, but deployment-imagescaler04.deployment-prep.eqiad1.wikimedia.cloud is very much running thumbor in a docker container. [01:30:06] https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Help has a list of all the things I found in a recent deplyment-prep instance audit [08:14:30] Morn. How could I find out where my DB is? :P https://wikitech.wikimedia.org/wiki/Phabricator says that Phab is on db1183, https://phabricator.wikimedia.org/T387174#10577932 mentions db1213. Thanks in advance! :) [08:15:16] andre: Hi, https://orchestrator.wikimedia.org/web/cluster/alias/m3 [08:16:53] andre: The first one is the master and the one that is being used. We change the master for mainantence issues "often" so it is "normal" that is not up to date [08:53:58] Ah, thanks a lot! Going to finally bookmark that [08:54:57] andre: you are welcome! [08:58:58] elukey: I left a reply for you on https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1084813 [09:05:00] federico3: ack I'll check later on! Is the patch ready for a review? [09:05:14] it is [09:05:19] super [09:05:28] I'll try to get a pass this morning [10:07:22] <_joe_> Emperor: given https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1121793 has been merged, I think it's time we consider rate-limiting thumbnails at non-standard sizes. It will take some time to get there but we should start collecting data as soon as the patch hits group1 (where commons is IIRC) [10:08:24] <_joe_> my take is we should only rate-limit cache misses at the CDN, but there's also an argument to only rate-limit requests that get to thumbor [10:08:35] <_joe_> but I'd like to hear your opinion as well [10:24:49] _joe_: Mm. Currently only deployed on beta, AIUI? But yes, I see this (and I think Amir.1 does too, but he's travelling) as a start on the road towards only generating standard-size thumbs at all. [10:25:37] If the best glide path towards that goal is a rate limit that tends to zero over time, I have no objections :) [10:26:22] From an infra-protection POV thumbor is the place to put the limit ; from a user education POV, the CDN is [10:41:46] federico3: I left some comments but mostly nits/suggestions-for-future, if the cookbook works and it has been tested I think that it should be ready for prime time [10:42:03] lemme know what you think about the try/except bits, I'll leave the final choice to you [11:02:52] my gut says that adding the rate limiting to thumbor itself makes more sense in light of us purging the old thumbnails - *but* thumbor's poolcounter implementation is broken enough that I think it'd be fairly dangerous to do so without an overhaul [11:15:27] "yay" [11:22:14] fixing poolcounter has been high priority for a year now but I dunno if it's gonna be looked at, especially if we're mulling over what replaces it [11:26:37] elukey: thanks, I've updated the CR. Speaking of unhandled exceptions, if a host is not known to puppet I'm getting a parsing error in the cumin query it seems https://phabricator.wikimedia.org/P74041 [11:27:53] yes, just put the fqdn part of the query at the end, the aliases are not empty [11:29:37] +1 I was about to say it [11:53:15] volans: you mean query = "A:db-all and not A:db-multiinstance and P{%s}" % fqdn ? [11:58:30] yes, correct [11:58:35] sorry lunch I can add more context later [15:07:12] volans: how can I acquire a spicerack lock without a cookbook? [15:25:22] ah I see Spicerack().lock()... [15:28:37] federico3: what do you mean "without a cookbook"? [15:28:59] without using a cookbook [15:29:11] you can't :D [15:29:22] what are you tying to do? [15:30:26] for usage in spicerack itself and the cookbooks https://doc.wikimedia.org/spicerack/master/introduction.html#distributed-locking outlines the 3 different ways locks can be acquired for the 3 different use cases [15:31:15] I can import spicerack, call Spicerack() and then lock() , no? [15:31:27] depends, where? [15:32:27] spicerack has a lot of configuration for all the other bits that imports and is not meant to be imported elsewhere being itself the automation framework [15:34:27] so I cannot use it in a library? [15:34:55] what's the specific use case? [15:35:07] without info I can't answer correctly ;) [15:40:27] acquiring, holding, releasing a distributed lock from libraries in our repos, in Python [15:41:25] FIRING: SystemdUnitFailed: prometheus-mysqld-exporter.service on db1253:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:43:53] there is no general purpose distributed lock as a service service. Spicerack locks are meant to be used by spicerack and the cookbooks as part of the framework. Also if as I suspect you're trying to add this to dbctl, spicerack depends on conftool, so making conftool depend on spicerack would create a circular dependency of imports. [15:44:49] the closest thing that handles locks is https://www.mediawiki.org/wiki/PoolCounter but it requires to keep a connection open for the duration of the lock and was deemed not suitable for spicerack given the long-duration of some cookbooks, hence why it has its own implementation of locks [15:45:54] if this is for dbctl I suggest you to discuss options with the conftool maintainers on the phab task to agree on a solution first. [15:48:21] this is a more general need for dba automation, dbctl would be one of the usecases [15:50:53] I'd be happy to discuss options if there is a task (or elsewwhere) with some context/information about the needs. [16:16:29] there's already a design doc and I'll open a task for the datastore. Regardind dbctl would a dedicated cookbook work? It could be a temporary solution for us [16:20:25] yes, that's what I suggested in https://phabricator.wikimedia.org/T387209#10578808. Get the same lock in all the cookbooks that interact with dbctl. And then you could make an sre.mysql.dbctl-lock cookbook that just acquires the lock and wait for the operator to complete with ask_confirmation to cover the manual uses cases of I need to touch dbctl manually. [16:20:55] this of course would work best if you move all the automated dbctl usage to cookbooks, as right now dbctl is also run by a series of other scripts DP has around