[00:07:12] Could force a 404 return with Redirect or RedirectMatch. [00:07:49] (which if i matches first, will skip looking for directory index) [05:02:27] In 1h we are going to swap s7 master [05:17:09] I know I'm working too late when the DBAs are waking up and starting to do scary things. [05:22:45] haha [05:25:40] I'm in Eastern European timezone (UTC+2) and I just woke up fairly early, and you're already working, I don't understand how :P [05:34:23] Majavah: Just making sure everything is ready for you when you wake up! [05:34:24] :) [05:34:35] razzi: There's a puppet commit not merged, can I merge it? [05:35:45] Looks like it is just removing labsdb1012, so I am going to ahead and merge it, as I need to merge my change too for the failover [05:35:49] elukey: ^ [06:13:47] marostegui: ah snap yes +1 [08:08:13] hello people, if you have to puppet-merge please wait [08:08:20] there is an issue on puppetmaster1002 [08:19:10] elukey: I think the issue has been resolved. [08:19:21] In that light, we are starting the process for the k8s eqiad reinint [08:19:52] For interesting parties, https://phabricator.wikimedia.org/T277741 is the task [09:37:20] thanks legoktm [10:21:18] ok so I want to put into production our 2 new mc servers [10:21:54] adding 1 will not cause a key resharding, but adding the second will [10:22:11] (for memcached) [10:25:15] also, as far as redis goes, we will have one shard unavailable while a server is running puppet [10:25:28] I will add comments on the relevant patches [10:25:41] not issues are expected [10:49:14] <_joe_> Krinkle: probably by disabling DirectoryIndex in apache's configuration [10:50:06] <_joe_> I meant enabling [10:50:15] <_joe_> but I'm not sure that's what you want [11:55:13] I will postpone to memcached work for tomorrow so to make sure all our ducks are in a row :) [12:35:57] 🦆 [12:42:10] I think you mean 🦆🦆🦆🦆🦆 [12:42:35] 🚣‍♀️ [12:44:42] I don't think that many of them will fit in the rowboat [13:03:14] 🛥️ [13:48:34] lol [14:06:40] yes but then they are no longer ducks in a row(boat) [14:09:38] hmm, I'm pretty sure duck eggs are too large to count as roe [14:14:49] but they're raw [14:21:09] lol [14:23:29] anyone that's involved in my reporting chain, please forget that i started this ^ [14:24:49] i request a 1:1 with you in 5 minutes. [14:26:25] 🏃‍♀️ [14:55:49] _joe_: hm so directory index on and one sub directory not having an index file is considered an error? Or have we enabled some extra verbosity to get these? Seem like maybe we can just turn that off if the messages aren't useful. The volume is low though so not so important I guess. Just a bit noisy compared to the rest [14:55:59] We do use index files in a sub sub directory again [18:04:34] moritzm: small question on https://gerrit.wikimedia.org/r/c/operations/puppet/+/674392, is the expectation that switching each team's services to `logstash-deployment-03` is done by the service owners in separate patches or also in that same patch? [18:05:22] I'm imagining the former (separate patch) [18:34:34] ryankemper: everything not in that patch needs to be changed via horizon [18:36:51] it can be either in Horizon or in the repo [18:37:09] both ways would work [18:38:08] the values are currently set in horizon, and afaik horizon overrides operations/puppet [18:46:53] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Hiera#Precedence_of_hiera_keys [18:52:22] if an SRE runs something like "systemctl status sometimer" they get the status and a few log lines, if a non-SRE on the same machine runs it, they get the status but none of the log lines, which sucks to debug failed timers. both are NOT using sudo/root, so for a moment it's like "wtf, but works for me without sudo". The reason is though that we automatically give all members of LDAP ops also the [18:52:28] "adm" group and adm can read logs. so there is a need for reading those logs for non-ops shell users in other admin groups [19:05:04] ryankemper: wikidata-query as contained in the patch will be switched when I merge this tomorrow, but maybe there are additional settings per project settings in Hiera