[00:28:18] * theopolisme groans because labs is being grumpy [00:28:24] yay there we go [00:28:30] Silly login server [00:28:54] I spoke too soon :/ [00:50:40] Coren: ping? [00:50:41] -rwxr-xr-x 1 yuvipanda svn 1405 Aug 2 00:49 receiver.py [00:50:45] local-suchaserver@tools-login:~$ take receiver.py [00:50:47] receiver.py: You need to share a group with the file [00:50:50] what gives? [00:58:08] YuviPanda: Coren is dining. Shouldn't the group be local-suchaserver? Is the directory +...s? [00:58:19] it's just home [00:58:23] err [00:58:23] hmm [00:58:24] maybe [00:59:41] It is (assuming /data/project/suchaserver = drwxrwsr-x). How did you put receiver.py there? [01:02:05] scfc_de: I scp'd it to ~ for yuvipanda, then cp'd it there [01:04:23] YuviPanda: Hmmm. The latter cp should have honoured the s. [01:04:37] Care to repeat with another file? [01:04:48] yeas, give me a moment [01:05:25] scfc_de: scp'd receiver.py to ~ (yuvipanda) [01:05:32] did a mv [01:05:34] to /data/project [01:05:51] scfc_de: take still teslls me /data/project/suchaserver//receiver.py: You need to share a group with the file [01:07:18] Sure, the file's group is svn. "touch /data/project/suchaserver/test"? [01:07:25] (As yuvipanda.) [01:07:49] touch'd [01:08:24] Aha, cp and mv work differently. [01:08:29] oh? [01:08:49] ah [01:08:50] yes [01:08:51] cp works [01:09:06] Tested it with local-wikilint (touch local-wikilint/abc and touch def && mv def local-wikilint/). Well, fuck :-). [01:11:37] http://www.redhat.com/archives/k12osn/2007-February/msg00253.html suggests standard Unix behaviour? [01:15:32] hmm, alright [01:15:35] I wasn't aware [01:19:28] Neither was I :-). [01:19:46] scfc_de: saw https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Dynamic_http_routing [01:19:47] ? [01:27:00] No, but I listened in on your and Ryan's conversation re that. Sounds fascinating, however for Tools I don't know how "custom" we want individual tools to be. If every tool (or its framework) brings its own webserver, I fear a lot of people will want to use that "just 'cause I can" :-). [01:28:12] I'm sortof happy with that, really :P [01:28:21] gevent! gunicorn! ruby! go! rust! assembly! intercal! [01:29:06] it's also a big fuck you to apache and CGI :P [01:32:12] Great minds can accomplish great things :-), however looking for example at how people want their own instances because they think they hit some resource limit, I don't know if the majority of Tools developers should be given that freedom :-). [01:32:48] with enough guidance I bet we can make them do great things :) [01:32:57] apache can be relegated to just serving PHP, in the end [01:33:17] scfc_de: I guess a lot of them still have a 'toolserver' mindset, which I *guess* didn't have as much infrastructure [01:33:34] nor as much flexibility, nor was as open to volunteer participation in the *infrastructure* (puppet? spare instances?) [01:36:27] scfc_de: if I submit a job as myself (and not a tool user) - 1. is that allowed? 2. will that just get my previlages? [01:36:43] &ping [01:36:44] Pinging all local filesystems, hold on [01:36:45] Written and deleted 4 bytes on /tmp in 00:00:00.0006270 [01:36:48] sigh [07:19:27] I don't know; I am not afraid of those who speak Puppet or just know what that is :-). My concern are more people who see "the standard Apache doesn't allow this miniscule option that I don't really need, but I WOULD LIKE TO!!!eleven!!" and then roll out their own webserver. Lots of duplicated effort and wasted resources (*maybe*). [07:19:27] YuviPanda: Yes, I'm not aware of any prohibition, 2. yes. [07:19:27] scfc_de: hmm, that is true. Considering how there's already been an attempt to... rewrite memcached and syslog and the OOM killer and... [07:19:27] Written and deleted 4 bytes on /data/project in 00:01:28.2124880 [07:19:27] :-) [07:19:27] Slow much. [07:19:27] scfc_de: but still, that's okay, perhaps? what's the worst that can happen? [07:19:28] they can't run apache on the grid can they? [07:19:28] YuviPanda: Why not? [07:19:28] good point. they probably will. [07:19:28] ugh. [07:19:28] well [07:19:28] they actually can't [07:19:28] since we won't have apache on the exec nodes :P [07:19:28] YuviPanda: Local install :-). [07:19:29] Worst case is that instead of concentrating development on what the *tools* actually do, people will spend more time on the scaffolding -- which IMHO is the job of the Tools project. [07:19:29] true [07:19:29] but then a lot of the 'newer' languages have moved away from the PHP model, which is what apache is suited for [07:19:29] put files somewhere, you'll run things somewhere [07:19:29] Yes, and I see Coren's pain :-). [07:19:29] Coren is probably going to give up on uwsgi tyrant, and I can understand why. Forcing the 'reverse proxy' model into the older 'files in a location' model is hard [07:19:29] CGI is a massive waste of... everything :P [07:19:29] scfc_de: what we can do is to have the power available, but let things be hidden behind wrappers. [07:19:29] scfc_de: run a python job by doing 'websub' with these parameters, for example :P [07:19:29] scfc_de: also apache doesn't support websockets, and that sucks [07:19:29] I'd prefer "we support A, B, C", but we do *that* with a "promise". Re Python & Co., I lack the experience there, so I don't know what's needed for proper setup. [07:19:29] I don't think Coren can setup uwsgi in a way that's satisfactory to him [07:19:29] (BTW, JFTR your project will be valuable for instanceproxy alone.) [07:19:29] neither can we do node, go, scala, ruby, whatever. [07:19:29] scfc_de: true. we'll save precious ipv4 addresses! [07:19:29] And have arbitrary ports! [07:19:30] that too! [07:19:31] 20000 ft view on uwsgi: All apps are persistent "daemons", central distributor that forwards request to those? [07:19:31] scfc_de: pretty much, yeah [07:19:31] it's also true for rack (ruby), jack (nodejs), go (built in, IIRC?), jetty (scala / java), etc [07:19:31] even fastcgi runs like that [07:19:31] and supports lots of languages [07:19:31] apache-php is the anomaly here, since php is deeply integrated into apache [07:19:31] I saw the disdain for mod_wsgi, but can't we use that? Split the requests according to tools so not every server must run all apps (we already do that, BTW)? [07:19:31] I think it's not too well maintained, and Coren didn't like it [07:19:31] for other reasons too, IIRC [07:19:31] also nobody who runs python in production uses mod_wsgi anyway. [07:19:32] And what are the problems with uwsgi? [07:19:32] Or is that the wish to have all apps run as grid jobs? [07:19:32] that's just Coren trying to re-create the SGE + routing jobs with tyrant, I believe [07:19:32] :) [07:19:32] scfc_de: indeed, since they are *persistant* deamons, having them run as grid jobs makes sense [07:19:32] (to me, at least) [07:19:32] and to Coren too, since he talked about just using fcgi + spawning them out to the grid [07:19:32] Yes, but if the design stands in the way of success ... :-) [07:19:32] heh :P [07:19:32] let's see if Coren figures out a way [07:19:32] Just because we have a grid doesn't mean we need to run everything there. [07:19:32] scfc_de: sure, and I do not know the details of what issues Coren ran into [07:19:32] I'd be happy if we got native uwsgi support, and i'd be doubly happy if we got rid of apache :) [07:19:32] (Though it would be *really* cool.) [07:19:32] scfc_de: it'll happen! Killing the apaches for anything not php might be my long term goal :) [07:19:32] At Toolserver I had to endure Zeus, and I'm not sure that was an advantage :-). [07:19:32] what was Zeus? [07:19:32] my interactiosn with the toolserver were limited to running a few queries now and then for people :) [07:19:32] *Is*. http://en.wikipedia.org/wiki/Zeus_Web_Server [07:19:33] ugh not open source [07:19:33] scfc_de: shouldn't happen, I think. This will be based off nginx (which we use in production), lua (which we use in production / for users too) and redis (which we use in production) [07:19:33] and also puppetized + documented + open source [07:19:33] okay, 'documented' was stretching it :P [07:19:33] but the rest of it is true! [07:19:33] I don't have strong opinions on Apache; the problem on Toolserver was the knowledge people have about Apache and about Zeus :-). If we use something that's deployed in production, I don't see any problems. [07:19:34] yeah [07:19:34] scfc_de: our previous plans was to use hipache, which did make me a bit queasy since it's not in production [07:19:34] *our* production [07:19:34] nginx+lua+redis doesn't have that problem [07:19:34] Yep. [07:19:35] scfc_de: what exactly was zeus used for, btw? [07:19:36] YuviPanda: For http? [07:19:36] scfc_de: like, where did that fit in, with apache being around too? [07:19:36] I think River was a Solaris guy, and choosing Zeus over Apache was a side product of that. Maybe it is better integrated with Solaris (well, I doubt that, Apache should be fine as well), maybe he was more familiar with it. [07:19:36] oh, so it was used *instead* of apache? [07:19:38] Yes. [07:19:39] ow that's bad [07:19:39] (for php, at least) [07:20:17] Does anyone know how I can manually deploy changes on beta labs? [07:20:17] I tried updating /usr/local/apache/common and running scap, but that gives me a segfault in parsekit.so :( [07:20:17] (I need to do this because VisualEditor is broken and stuck on July 25th) [07:20:24] RoanKattouw, I think you just make changes on deployment-bastion and they get instantly deployed via glusterfs [07:20:24] at least that's according to the second-to-last line of https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Overview#Architecture [07:20:27] RoanKattouw, I assume it's fixed now? [07:20:27] yes [07:20:27] I ended up manually fixing the extensions.git repo, then triggering the Jenkins job my merging an extension commit [07:20:27] Though now we're having RL caching issues [07:21:05] @notify T13|sleeps [07:21:05] This user is now online in #wikimedia-mobile. I'll let you know when they show some activity (talk, etc.) [07:21:05] wtf [07:21:05] @notify T13|sleeps [07:21:05] :/ [07:21:05] !poof [07:21:05] addhome, T13|sleeps ping [07:21:05] !ping [07:21:05] !pong [07:21:05] !poang [07:21:05] !pang [07:21:05] !pung [07:21:05] !derp [07:21:05] ???????? [07:21:05] petan, what's wrong with wm-bot [07:21:06] *POOF* "Wadda need?" *POOF* "Wadda need?" *POOF* "Wadda need?" [07:21:07] !pong [07:21:08] pang [07:21:09] pung [07:21:10] derp [07:21:11] Don't mess with me. [07:21:12] This user is now online in #wikimedia-mobile. I'll let you know when they show some activity (talk, etc.) [07:21:52] someone restarted it absolutely pointlessly in a wrong moment [07:21:53] !ping [07:21:53] !pong [07:22:05] poor thing [07:22:24] it's back [07:22:24] :O [07:22:50] Pet an pong [07:23:07] He he that's me? XD [07:23:19] Blame t13! :) [07:24:43] lol [07:24:46] what were you doing?? [07:27:14] @notify T13|sleeps [07:27:14] This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.) [07:27:32] addshore: what happened?? #wm-bot is full of shit [07:27:37] I'll let t13 tell you :0 [07:27:37] he's not in here [07:27:37] I need to get breakfast ;p [07:27:37] Oh .. [07:27:44] Btb [07:27:51] :) [07:55:20] [bz] (8NEW - created by: 2Addshore, priority: 4High - 6enhancement) [Bug 48894] Include pagecounts dumps in datasets - https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [07:55:31] !ping [07:55:31] !pong [08:20:04] addhome, ping [08:20:40] pong [08:20:56] addhome, you sure know how to flood my inbox. :p [08:21:32] Cyberpower678 ?? [08:21:42] petan, ?? [08:21:50] you just joined #wm-bot, typed 2 non-existing commands and left o.O [08:22:02] petan, yhes. [08:22:15] :0 [08:24:02] addhome, I'm going to work on Peachy some more. I'm almost done the gut work. [08:27:42] :> [08:27:50] Cyberpower678: I want to try and catch you up ;p [08:41:29] addhome: you should migrated to tools-redis at some point [08:41:29] :) [08:44:43] YuviPanda: is it working the sames as -mc? [08:45:00] addhome: yeah, exactly the same... except it's got 7 times as much memory [08:45:02] and 4 cores :P [08:45:04] :D [08:46:06] ill do it in a second [08:46:12] once -login responds [08:48:22] addhome: <3 [08:48:26] &ping [08:48:27] Pinging all local filesystems, hold on [08:48:28] Written and deleted 4 bytes on /tmp in 00:00:00.0007550 [08:48:29] Written and deleted 4 bytes on /data/project in 00:00:00.0097340 [08:48:36] just srated working ;p [08:52:08] YuviPanda: switched [08:52:14] :D [08:52:15] works fine? [08:52:29] seems to be [08:53:04] hehe YuviPanda http://ganglia.wmflabs.org/latest/graph.php?r=20min&z=xlarge&h=tools-redis&m=load_one&s=by+name&mc=2&g=cpu_report&c=tools [08:53:06] \o/ [08:53:50] addhome: heh, looks very overloaded :P [08:54:21] haha :p [08:54:25] cheers YuviPanda ! :) [08:55:46] addhome: :D [08:57:23] addhome, I spent all morning reading my emails. 92% came from you. :p [08:58:34] :O [08:58:44] it was only about 8 commits or somethin xD [08:58:46] *something [08:58:54] grr nickname [08:59:38] Received 39 emails from Addwiki [08:59:50] addshore, ^ [09:00:13] addshore, make that 40 now. :p [09:00:19] addshore: are you coming to wikimani? [09:00:37] YuviPanda, where it is take place this time? [09:00:42] Hong Kong [09:15:29] YuviPanda: no worries not going to rewrite apache today [09:15:37] :D me neither [09:16:10] but there is a good point in rewriting memcache (which is pretty simple thing) in c# and making OOM killer as external service, not included in kernel and more flexible [09:16:28] it's not reinventing a stone wheel, it's changing a stone wheel into Pirrelli tires [09:17:28] well, if you think rewriting memcached is 'simple', please read https://www.varnish-cache.org/trac/wiki/ArchitectNotes [09:17:37] Redis is the tires. [09:17:58] well redis is nice, but it's still missing important things [09:18:33] like authentication... also did you check if redis is same fast as memcache? did you run any kind of benchmark before you decided to force all tool developers to move from that to redis? [09:18:57] I'm evil, don't you know? :) [09:19:22] it's possible to secure keys in redis. not possible in memcached. end of story. [09:20:04] and did you even click that link? [09:21:22] it's not possible to secure key in redis neither in standard memcached. that is end of story [09:21:33] redis has exactly same way how to secure keys [09:21:38] which is none [09:22:05] YuviPanda: yes I clicked that link. The guy who wrote that is... cool, people should learn from him, especially modern python programmers [09:22:26] anyway, that what he say is not entirely true, you can easily bypass swap and disk caches on linux kernel [09:22:30] :) ok [09:23:01] raw devices == no cache, (swappiness == 0 || no swap) == no problems with swapping [09:23:12] it's really a problem of architecture [09:23:14] have a look on oracle [09:23:30] it does exactly the same what squid and yet is considered the best database system on world [09:23:33] which is likely true [09:23:42] yeah, one that can be solved by writing low level tools in a language with non-deterministic GC + object overheads :) [09:24:13] I don't really know if memcache is a low level tool, but in this you are right [09:24:38] using c++ would be better, but that would mean spending 100 times more time to make it [09:25:20] indeed. [09:25:40] also having any form of required auth up front would kill most of the perf stuff anyway. [09:31:28] not necessarily, you could open a session, auth, and THEN you would do the perf stuff [09:31:40] YuviPanda: no [09:31:46] a w :( [09:31:46] but of course most of python kids prefer to open new session for every single key [09:32:01] because then the source code looks more cute even if it suck [09:32:17] hehe python kids :P [09:32:40] just saying petan. Insulting everyone who writes python code isn't bound to . [09:32:56] help make friends [09:32:59] or do anything productive [09:33:05] I am not insulting everyone, I am just insulting kids :P [09:33:12] yeah, keep doing that [09:33:17] * YuviPanda goes to do something productive instead. [09:34:19] you are taking stuff too much seriously... [09:35:20] why in the world should you open a new connection to redis 100 times if you wanted to insert 100 keys? [09:35:34] that is performance killer as well [09:44:52] anyone know where the repo / source is for http://tools.wmflabs.org/geohack/ ? [09:45:00] oh, wait... [10:06:46] there is twitter memcached proxy [10:06:53] Twemproxy is the name [10:07:56] you install it on the frontend server and it open a single connection to each cache server in which the requestes are multiplexed [10:08:02] i think we got it in prod [10:15:37] hashar: yeah, we run twemproxy in prod :) [10:16:50] hashar: I think we also now use Redis for sessions, not memcached. Since REdis does cross datacenter replication [10:19:24] yup sessions are in redis [10:19:33] not sure whether cross dc replication was the killer feature though [10:20:15] lack of problems with restarts + cross dc replication, I suppose? :) [10:20:43] we also use Redis for job queues, and I was talking to ori about the IRC RC changes thing and redis will probably be involved there in some form too [10:20:50] hey jarry1250! Will you be coming to wikimania? [10:22:12] one day I will reengineer our job queue system [10:22:15] it is hoooriiiibbble [10:23:53] hashar: at least we aren't using mysql as a queue :P [10:31:50] YuviPanda: well that is just the datastore [10:32:14] YuviPanda: my complain is more about the evil PHP scripts under maintenance/ and the VERY lame jobrunner shell script :] [10:32:22] hehe true truie [10:32:26] one step at a time, I guess [10:32:29] maybe next year I will attempt a proof of concept [10:32:33] nice! [10:32:39] I need to phase out jenkins first [10:39:34] hashar: phase out? [10:39:44] "get rid of" [10:39:45] :D [10:39:58] and replace it with? :D [10:40:04] the rough idea would be to have Zuul enqueue job request in some system [10:40:12] then have workers fetching the job requests and handling them [10:40:21] docker! [10:40:22] using http://gearman.org [10:40:25] a job server [10:40:33] (for isolation, at least0 [10:40:34] ) [10:40:38] yup [10:40:55] and then [10:41:21] rewrite MediaWiki/Wikimedia job system to send jobs requests to gearman [10:41:50] :) [11:00:25] * T13|sleeps yawns [11:05:49] andrewbogott_afk: ping [11:05:54] @notify andrewbogott_afk [11:05:54] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [11:42:34] &ping [11:42:34] Pinging all local filesystems, hold on [11:42:35] Written and deleted 4 bytes on /tmp in 00:00:00.0005830 [11:43:52] Written and deleted 4 bytes on /data/project in 00:01:17.8795490 [11:50:31] addshore ping [11:50:34] can you try ssh to bots-labs [11:50:52] or Damianz [11:51:11] I disabled gluster mount for /home there [11:51:18] so I would like to figure out how it works [11:51:23] * whether it works [11:53:12] (03PS1) 10Yuvipanda: Refactor subscriptions.py [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77291 [11:54:44] (03PS1) 10Yuvipanda: Move i18n extensions to #mediawiki-i18n [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77292 [11:57:13] (03PS2) 10Yuvipanda: Move i18n extensions to #mediawiki-i18n [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77292 [11:59:57] (03CR) 10Yuvipanda: [C: 032 V: 032] Refactor subscriptions.py [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77291 (owner: 10Yuvipanda) [12:00:51] (03CR) 10Yuvipanda: [C: 032 V: 032] "siebrand says okay!" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77292 (owner: 10Yuvipanda) [12:07:08] (03PS1) 10Yuvipanda: Fix missing comma [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77295 [12:07:09] (03CR) 10Yuvipanda: [C: 032 V: 032] Fix missing comma [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77295 (owner: 10Yuvipanda) [12:07:16] heh, my shame! [12:12:23] I made a number of changes to the caching behaviour of the NFS server that may mitigate the negative effects of controller stalls. Please keep an eye on things and see if the pain is less. [12:13:03] &ping [12:13:03] Pinging all local filesystems, hold on [12:13:04] Written and deleted 4 bytes on /tmp in 00:00:00.0006270 [12:13:05] Written and deleted 4 bytes on /data/project in 00:00:00.0065830 [12:13:26] (i.e., I can't do anything against the controller wedging but I think I made it so that it doesn't affect thigs as much or for quite as long in many cases) [12:14:47] Coren: did the firmware update do anthing? [12:14:50] *any [12:15:31] That requires a long downtime; we're not going to do this until we switch to the backup server. [12:15:38] oh, right [12:16:11] ([bleep bleep] firmware update requires an idle controller and booting off a special image) [12:18:05] man, they should all just come off a 'firmware app store' :P [12:18:07] * yuvipanda runs away [12:20:57] [bz] (8NEW - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 52452] Have public, readonly, up-to-date git repositories available for all tools to use - https://bugzilla.wikimedia.org/show_bug.cgi?id=52452 [12:32:21] petan: [12:32:24] pong [12:32:31] yes [12:32:35] can you try ssh to bots-labs [12:32:49] I disabled gluster there and want to check if keys still work [12:33:03] I also had to umount /public [12:37:59] addshore? [12:38:13] *tries now* [12:38:33] Coren: is one of side effect of your tweaks that it takes horribly slow now for files to sync between servers? [12:38:49] "sync" [12:39:01] Coren: my tool is writting to a file and I see the changes on other server after a minute [12:39:14] petan: That's... not how NFS works. [12:39:20] ? [12:39:22] petan: I can get in fine [12:39:27] addshore: cool [12:39:46] naturally possible dns spoofing and remote host id changing :p [12:39:47] (03PS1) 10Yuvipanda: [WIP] Store canonical info about subscriptions in mysql [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77300 [12:39:50] but *ignores* [12:39:54] petan: The buffering happens in your tool, NFS does not, /cannot/ have lag of the sort you describe. [12:40:17] Coren: nope, I did "execute some tool >> logfile" [12:40:22] it produces output in realtime [12:40:25] http://ganglia.wmflabs.org/latest/?c=tools [12:40:30] but logfile is being appended with horrid lag [12:40:32] what the .... happened an hour ago?? [12:41:18] petan: 99.99% of programs's standard output is buffered unless you took great care to not buffer it. [12:41:35] programs'* are* [12:41:51] ok but that is /bin/dash what is buffering it then [12:41:58] meh [12:42:04] why it happens only on tools project? [12:42:05] libc. :-) [12:42:15] I have done a same thing on bots-labs with wm-bot and it works fine [12:42:24] I can tail -f the file and I see it real time [12:42:33] but on tools project it takes a minute to refresh [12:43:00] don't tell me it's because tools project has different libc [12:43:32] petan: The events you describe are not consistent with the way filesystems work, nor NFS. Compare with a tail on the same host where the output is taking place. [12:44:15] yes I am talking about this, I am doing in on localhost on bots-project but not on tools of course [12:44:23] regular users aren't able to ssh to to exec nodes to do that [12:44:35] I see similar behavior on all my logs, I am pretty sure that's just buffering from the logging framework I use [12:44:41] you don't want an fsync after every write [12:44:46] I do [12:44:53] :P [12:45:00] yuvipanda: Well, you might want it in some situations. [12:45:07] Coren: sure, but not by default [12:45:20] yuvipanda I don't really care about fsync I want my tool behave same on tools project as it is on bots-project [12:45:23] for *logging* [12:45:49] I am trying to figure out how to accomplish this on tools [12:46:05] it was quite easy on bots... [12:46:09] petan: If you have different behaviour, then you have different circumstances. [12:46:14] ? [12:46:44] only different circumstance is that the project changed... [12:47:02] well, on bots I was using local filesystem not a nfs [12:47:07] hehe [12:47:15] that's another circumstance that changed [12:47:22] and quite significantly worsen stuff [12:47:39] petan: POSIX filesystem semantics have very well defined semantics, if you rely on unspecified behaviour, then you get unspecified results. [12:48:29]