[08:25:25] Could I reboot one of those hosts to look at the NIC Bios options? https://phabricator.wikimedia.org/P12989 (for T250367) [08:25:25] T250367: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367 [08:28:13] XioNoX: yeah one thanos-fe host can be depooled at any time and inspected [08:30:21] godog: thanks! [08:31:12] np [09:40:43] Hey all! I'm doing a refactoring workshop tonight (7pm UTC). It's going to be Python based for once: https://www.mediawiki.org/wiki/Code_Health_Group/projects/DevEd/Workshops#Refactoring_Workshop [09:40:50] ping me if you want to join [10:03:36] 11 pm for me... too late in the evening [10:03:48] will there be a recording or anything lke that? [10:10:11] <_joe_> yeah I have two hours of meetings starting at 8 pm tonight, I'll pass too :/ [10:16:40] I'm doing those workshops monthly. There's an EU friendly one as well. Have a look at the page above. [10:20:28] ah I was excited for the python-ness... well, I'll keep my eyes out for next month [10:49:01] gehel: can I get an invite? [10:49:13] if I can, I will join [10:49:54] I will put down a reminder, nvm, :p [10:53:23] effie: {done} [10:53:32] <# [10:53:33] <3 [14:09:17] jbond42: is there a difference between `lookup(x)` and `lookup(x,)` in puppet? [14:12:52] kormat: i don't think so but honestly the later form is not one i have seen often so not sure. testing localy they seem the same [14:13:22] ok, cheers. the latter was probably unintentional, so i'll fix it [14:14:08] ack thx [14:40:23] jbond42: is PCC trolling me? https://puppet-compiler.wmflabs.org/compiler1003/25888/db1117.eqiad.wmnet/index.html [14:40:33] - port => 3323 [14:40:33] + port => 3323 [14:41:04] ahh. it used to be a string. now it's an int. [14:41:16] yes exactly that [14:41:16] Also, tabs vs spaces [14:41:21] because who needs types anyway [14:42:15] tabs are a pox on the world [14:42:41] * klausman slowly slinks out before the flame war starts [14:43:36] [ to be clear, the tab keyboard key is useful. Just not the tab character in source code, as in ascii character number 9 ] [14:43:50] flame suit ready! :) [14:46:56] wtf is going on with Etherpad? I keep getting "You sent too many messages to this pad so it disconnected you." and I'm not sending more "messages" than usual [14:47:27] that happened to me recently as well [14:47:45] even for moderately slow typing [14:48:58] yeah I've been frustrated by it as well [14:49:42] it's like, the client-side JS doesn't try very hard to batch characters into sending transactions (because doing so would probably result in some janky typing latency?), and there's a very not-fast-typing-friendly ratelimit interacting poorly with whatever batching might be going on [14:50:26] I'd guess there's probably a narrow window of typical typing speed that some of us happen to routinely hit, and if you go faster or slower than that window you don't see the problem. [14:51:43] (either that or they do no batching at all, in which case anything faster than X will trip it) [14:53:53] I guess the recent update introduced it, the release notes for https://github.com/ether/etherpad-lite/releases/tag/1.8.5 mention: [14:53:54] IMPORTANT SECURITY: Rate limit Commits when env=production [14:54:14] V(o)-LAN(s) tried looking into that already and we where not able to reproduce. It's definitely not fast typing or copy & paste things they use this combination of duration and "points" https://github.com/ether/etherpad-lite/blob/ceb09ce99ad080ae804d322804b22e3e8f0be005/settings.json.template#L495 [14:54:27] there is also a bug upstream fwiw [14:54:46] volans: there's bugs everywhere [14:54:48] and the etherpad's master was made aware :) [14:54:49] seriously volans?! I really tried :D [14:56:23] hahaha [15:03:46] <_joe_> I think we should just ditch etherpad for good [15:04:12] <_joe_> and use the etherpad mediawiki extension instead [15:04:57] <_joe_> how's that called, Liquidthreads? [15:13:13] We have numerous :D [15:27:10] bblack: they do a lot of batching, you just don't type fast enough for their batching algorithm to batch all of your typing as 1 batch :P [15:28:08] jokes aside, that's true ^. They do operational transformations and send "changesets" to the server. If you type a lot of stuff quickly it will be sent as 1 batch, but if you pause and linger you are almost definitely gonna get rate limited [15:28:56] this happens btw because they rate limit based on the IP and guess which IPs etherpad sees [15:46:04] hey :) I was looking for the mw + k8s channel but it seems like it's empty? I probably got the name wrong [15:52:44] mszabo: Hi! welcome. It's https://grafana.wikimedia.org/d/NQO_pqvMk/push-notifications?viewPanel=20&orgId=1&from=1602201152770&to=1602403260466&var-dc=eqiad%20prometheus%2Fk8s&var-service=push-notifications [15:52:46] sorry typo [15:52:49] it's #wikimedia-mw-on-k8s [15:53:02] the keys are so close together [15:53:17] lol [15:53:22] it's middle-click vs ctrl-v [15:53:25] oh it's ratelimiting on cache IPs? [15:53:39] can we make it sensitive to X-Client-IP? [15:53:50] bblack: I think it's worse, it's ratelimiting on the local envoy proxy [15:54:20] they already have a patch for the next version. They are creating some HTTP header indeed, hopefully configurable [15:56:30] akosiaris: thanks, I had the name without wikimedia- prefix :D [16:00:15] mszabo: yw [16:00:22] bblack: I think they got it fixed in https://github.com/ether/etherpad-lite/commit/ceb09ce99ad080ae804d322804b22e3e8f0be005 [16:00:45] not sure what clientIPAddress is in that context, it will require some more reading of the code. But it's not released yet [16:01:28] it's a bit weird because I am not sure what remoteAddress[client.id]; means in the context of etherpad [21:51:34] wkandek: effie: Hi, this isn't urgent, but just fyi that this is something I have time for this quarter - https://phabricator.wikimedia.org/T263494 [21:54:15] oh that is lovely, we had discussed beginning of Q1, we should have a serious look at it [21:54:18] thank you !