[05:35:09] <_joe_> no [08:26:31] <_joe_> I have a systemd-related doubt [08:26:35] <_joe_> say I have set [08:26:49] <_joe_> LimitNOFILE=65536 in a systemd unit [08:28:06] <_joe_> that unit launches a process that executes a child process - will that child have that number of open files available? [08:32:01] _joe_: I would assume any child would inherit that from its parent [08:32:18] same [08:32:20] <_joe_> this is an interesting problem, so the code I'm looking at uses os.execl(), which should indeed do that [08:32:38] <_joe_> but then it's executing a bash script [08:32:48] <_joe_> which I could cut off [08:33:30] it should inherit the limit but not concur to the same limit of the parent ofc [08:33:59] if its executing a bash script why not calling ulimit -n [08:34:11] to see which limit does the bash see [08:45:25] <_joe_> fsero: yeah I just did something like that, and yes, the limit is indeed correctly inherited [10:07:47] ema: [10:07:48] https://gerrit.wikimedia.org/r/c/operations/puppet/+/529927 [10:07:56] is the same as the one yesterday but for eqiad [10:08:11] any issue if we merge it now? [10:19:40] fsero: nope [10:19:46] ty ema [13:55:51] hello! I'm going to upgrade and reboot cr2-eqdfw in the next 20-ish min, no production impact expected, but some possible alert noise [13:59:19] ack! [14:01:55] tx XioNoX ! [14:19:02] time for the reboot [15:20:20] alright, doing the same with cr2-eqord [15:49:28] volans: i've documented our great hack on wikitech -- thanks for the idea :) [15:51:07] cdanis: lol, thanks! [16:00:13] cr2-eqsin upgrade it going to take a bit longer, because copying a 2G image file over there takes between 1 and 2h [16:00:36] ouch [16:00:43] is it scp? [16:02:31] I seem to vaguely remember that many years ago there were patches to scp to make it perform better on high-BDP links, and I seem to remember checking back more-recently after the passage of many years and finding they're still not in upstream and scp is still horrible on high-BDP :/ [16:04:12] I'm trying two options, one http from juniper to mr1 (~45min expected), and another http from install1002 to cr2-eqsin (~2h15 expected) [16:04:35] single stream + high latency [16:04:56] yeah [16:05:30] a single stream of TCP *can* be pushed at full bandwidth over high latency, but usually requires application-level support (code and/or config-tuning) to not suffer. [16:05:58] (more buffers, and/or depending on the applayer protocol, more applayer data allowed in flight before a round trip ack) [16:06:28] that's the problem with SCP. it rides on SSH, and SSH's protocol is too ack-happy to deal well with the BDP. [16:07:31] do we have aria2 available on say the eqsin bast host? [16:07:41] you could possibly hackaround this by splitting the file into several smaller chunks and scping those in several parallel sessions to one of the linux hosts in eqsin first, then reassemble and upload locally. [16:08:20] cdanis: I don't think so, but it might be installable! [16:09:26] "man split" for *nix CLI to split a file into a bunch of smaller files for xfer. [16:09:45] (nobody ever thinks about that one these days. it was handy in the modem days, and rarely in situations like this in the modern world) [16:10:47] I was hoping to not need to split because HTTP range requests might be enough, but TIL that most http/1.1 servers ignore that? [16:10:48] actually I stopped install1002 -> cr2 as it was taking multiple hours, and I'm now running scp from my laptop to cr2 (via bast5001), seems to be doing better (~30min) [16:10:59] "split -b 100M imagefilename" makes a bunch of files named xaa, xab, xac, ... [16:11:37] ah if you've got it down to 30 minutes it's probably not worth hacking on something more complicated :) [16:12:41] bblack: also because we don't do ssh agent forwarding, how could we copy the file from bast to cr ? [16:13:29] maybe through the mgmt interface + temporarily enable root password, but that's probably slow as well [16:13:38] I would temporarily run an http server on a high-numbered port on the bast host >_> [16:13:42] yeah [16:13:53] nc is your friend :) [16:13:53] unless juniper can only fetch via scp [16:14:03] juniper can do http [16:14:18] can nc? :) [16:14:21] why would it take 2h15 for an http stream though? that doesn't make sense [16:14:34] yeah that seems quite high to me as well [16:14:41] cdanis: but then that means having to allow that port on the strict router loopback filter :) [16:14:41] I think the original 2h15 was trying to scp halfway around the world straight to juniper [16:14:51] why would that be that slow? [16:15:16] because scp is awful over high-latency links historically (and even if some modern ones are fixed, I bet the juniper implementation isn't) [16:15:30] XioNoX said "http from install1002 to cr2-eqsin (~2h15 expected)" tho [16:15:35] 2h was pulling the file from cr2 and install1002 via http [16:15:52] `file copy http://install1002..... /var/tmp/` [16:15:55] http shouldn't have the same problems with bdp [16:16:07] oh it was via http [16:16:14] but yeah, still [16:16:15] junos implementation of http is probably very basic as well [16:16:30] that wouldn't have to do with anything [16:16:38] as long as it lets the tcp window size expand it should be fine [16:16:43] which it may not! [16:19:37] I suspect a bottleneck somewhere else [16:19:51] i.e. not in the junos tcp/ip stack [16:19:57] could be [16:20:08] going over the mgmt network or something like that [16:20:33] isn't the mgmt network still 1Gbps? [16:20:41] or it is something awful? [16:20:50] going over the oob link I meant [16:21:05] which is not 1G, no [16:21:15] ah yeah, could be that! [16:23:21] still even then, if there weren't protocol/software issues with delay, even a 100Mbit circuit should be able to send a 2GB file in ~3 minutes. [16:23:43] I think, if my brain isn't malfunctioning [16:23:49] did we even get a 100Mbps circuit? [16:25:04] I'm pretty sure it's physically 100Mbps, but I donno about soft limits maybe. [16:26:01] oh wait, I found the order form [16:26:10] it may be capped as low as 2Mbps heh [16:26:35] there's no units shown, but it does say "bandwidth burst cap: 2" on a 100Mbps circuit :P [16:26:41] I don't think it's a mgmt link issue as all the other MX204 file transfer took a few minutes [16:27:10] XioNoX: is the xfer over oob? [16:28:50] bblack: on the MX204 it's harder to know exactly where the packets are coming out from, but I don't think so [16:29:12] and if it did, ulsfo MX204s should have been as slow [16:29:54] a fetch from cr2-eqsin shows exactly a speed of 256 KB/s == 2Mbps fwiw [16:30:08] gotta run to a meeting now, ttyl :) [16:30:30] fetch to where? [16:30:44] just fetched junos from install1002 [16:31:20] also it started very fasts but then dropped at a few Kbps [17:13:08] alright all router upgrades are done for today! Appologies for the stray page