[00:51:19] andrewbogott_afk: Did you see the ipad lock screen wallpaper I did? [01:24:20] !log adding myself to analytics project to check DNS addresses and add Dario and others [01:24:21] adding is not a valid project. [01:25:57] !log analytics adding myself to analytics project to check DNS addresses and add Dario and others [01:25:59] Logged the message, Master [01:29:42] !log analytics adding bsitu, kaldari [01:29:44] Logged the message, Master [01:40:12] adding DarTar failed for some reason [05:25:09] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 0 minutes) [05:38:34] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 13 minutes) [05:51:59] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 27 minutes) [06:05:21] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 40 minutes) [06:18:50] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 53 minutes) [06:32:16] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 67 minutes) [06:45:40] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 80 minutes) [06:59:01] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 94 minutes) [07:12:23] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 107 minutes) [07:25:48] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 120 minutes) [07:39:14] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 134 minutes) [07:52:44] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 147 minutes) [08:06:15] Warning: There are 1 users waiting for shell, displaying last 1: DerekFaiers (waiting 161 minutes) [08:12:03] that's taken care of, then [08:24:22] yay :D [08:24:55] btw addshore [08:25:01] u here? :D [08:25:07] @notify addshore [08:25:07] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [08:31:12] petan: he'll show up probably like 6-8 hours from now [08:31:30] why u think si [08:31:31] so [08:40:47] petan: based on his pattern in general [08:40:57] :0 [09:00:40] !log deployment-prep Updating puppet repositories on search01 and searchidx01. Running puppet on both of them. [09:00:43] Logged the message, Master [09:22:50] Warning: There are 1 users waiting for shell, displaying last 1: Prolineserver (waiting 0 minutes) [09:36:16] Warning: There are 1 users waiting for shell, displaying last 1: Prolineserver (waiting 13 minutes) [09:49:46] Warning: There are 1 users waiting for shell, displaying last 1: Prolineserver (waiting 27 minutes) [09:50:38] * addshore pokes petan [09:51:05] haha Jasper_Deng_away you say 6-8 hours :P, make that 1:20 ;p [10:03:20] Warning: There are 1 users waiting for shell, displaying last 1: Prolineserver (waiting 40 minutes) [10:24:55] addshore: btw if you noted that there aren't many preferences in my client, it's because I didn't make a layout to change them, but there is ton of them in config file (misc -> config) [10:30:18] hehe kk :) [11:50:13] is there any documentation on role::mediawiki-install::labs ? [11:52:05] !log deployment-prep Successfully added Mark Bergsma to deployment-prep. [11:52:09] Logged the message, Master [13:01:48] @notify andrewbogott [13:01:48] I will notify you, when I see andrewbogott around here [14:13:49] New review: Mark Bergsma; "This is a very different workflow from how the rest of Labs works, and I'd like to keep that consist..." [operations/puppet] (refs/meta/config) C: -2; - https://gerrit.wikimedia.org/r/59162 [14:45:04] !log tools reboot -login (NFS mount woes) [14:45:05] Logged the message, Master [14:48:27] andrewbogott: Heyas. [14:48:53] 'morning! [14:50:09] Coren nfs mount? [14:50:30] petan: Shortly. :-) [14:50:39] what for [14:52:08] petan: Death to gluster [14:59:48] I remember people were making fun of me using nfs instead of such cool things like gluster on labs [15:00:12] Oh, gluster is very cool. It also sucks very much. :-) [15:02:26] Oh, ffs. gluster *had* to break, didn't it? [15:02:39] * Coren wasn't really asking for a demonstration. [15:19:16] Coren, what broke? Do you want me to troubleshoot? [15:19:36] andrewbogott: Odd interaction with autofs. Fixed by a reboot. [15:19:50] 'k [15:20:04] andrewbogott: If you have a bit today, though, I could use a hand with adapting the project fs export script for labstore3 [15:20:54] Sure, I'm at a good spot to change tasks… [15:20:56] You're talking about manage-volumes-daemon? [15:21:07] * Coren nods. [15:22:33] OK… it's pretty gluster-heavy as is so we should probably fork it. [15:23:02] Do you want to email me a crib sheet with the commands we need to e.g. create/delete volumes, provide instance access etc? [15:23:19] Or do you already have a draft of the script in the works? [15:25:37] andrewbogott: Well, the fun part is that because of the snapshots it has to also maintain that aspect and (possibly) change the exports hourly; the manage snapshots part is done with a (perl, atm) script that mounts all the stuff locally. Creating volumes, and exporting them is left to do, but because of the snapshot the exports have to be revised hourly. [15:26:12] I have the resulting exports file done; the pattern is simple, and I'm using /etc/exports.d/ per volune [15:26:58] I think I need background about 'snapshots' -- what is that? [15:27:19] db snapshots, volume snapshots, tool snapshots...? [15:27:41] volume snapshots; I maintain a snapshot of the past 3 hours, 3 days, and 2 weeks. [15:28:33] Go look at the mounts on labstore3 if you want to see [15:28:46] ok. And that requires the exports to be changed because... [15:28:57] the formerly live volume becomes the snap and we create a new live volume? [15:29:24] No, but the set of available snapshots changes every hour. [15:29:41] And, AFAIK, it's not possible to put wildcards in exports. :-) [15:29:46] Oh! OK, so its the snapshot exports that update, that makes sense. [15:30:09] So an instance has read-only mounts to an ever-shifting set of snaps? [15:30:18] Will that be every instance or just for tools? [15:30:53] Every instance, the snapshots are of the entire storage volume; the instances get an export of a subtree of those snapshots. [15:31:27] The live fs is /srv/$project/{project,home} [15:31:46] The snapshots live in /time/$timestamp/$project/{project,home} [15:32:05] (Well, strictly speaking, those are the *exports*) [15:32:11] This sounds great! But… doesn't it require 10x as much storage space as we were using before? Is there really that much slack? [15:32:19] (The live fs is /srv and the snapshots are /time/$timestamp) [15:32:40] andrewbogott: The snapshots are COW, and deduplicated. [15:32:49] Oh, OK. [15:33:02] (And they are set readonly, so not COW either for real) :-) [15:33:46] In practice, the storage requirement = total storage used + total difference since oldest snapshot) [15:34:00] Sure, that makes sense… and probably isn't very much in most cases. [15:34:35] I'd expect not. If things get out of hand we might have to reduce the number of weeklies we keep, but for <1 week is should remain managable. [15:34:59] So having the daemon refresh every export on every run… probably should be fine. None of that is very expensive as far as I know. [15:35:30] It's basically just a parse + a copy to kernel space. [15:35:47] How often does it run? [15:35:59] I'm checking... [15:37:14] 3 mins. [15:37:27] (A completely arbitrary choice.) [15:37:53] Heh. If we want to be a bit smart about it, we can actually check that nothing changed and just not redo the exports. [15:38:30] * andrewbogott nods [15:46:29] Coren: So you already wrote the snapshot stuff but not the volume-creation stuff? What, then, are you taking snapshots of? :) [15:47:09] Strictly speaking "volume creation" is now just "mkdir in the single filesystem" :-) [15:47:30] It's the exporting that's the fun part. [15:48:09] Oh… so is this multi-tenant, or is security entirely managed by the exports file? [15:49:08] security is managed by the exports file; Ryan didn't want to have multitenancy since there is no performance advantage (and no small amount of hairy mounts if we had to have snapshots per project) [15:50:02] OK, and I take it there's no concept of starting/stopping volumes in this case [15:50:11] * andrewbogott is looking at what to prune [15:50:36] Also the current daemon has some logic to run commands on each node of the cluster. Not needed for NFS, right? [15:50:55] Right. [15:51:30] So all the daemon will do is: [15:51:31] a) mkdir [15:51:38] b) update exports [15:51:53] c) log orphan volumes ( <- this is what we do instead of deleting them) [15:52:17] And there's a single system-wide export file? Where is it? [15:52:45] I'm using /etc/exports.d atm, dunno if we want to keep that or make one big /etc/exports instead. [15:53:17] Ah, with one file per project? [15:53:22] That's better. [15:53:35] Seems simpler to me [15:53:55] There's some trickery we need to do with nfs options, fsid in particular, but it's simple. [15:54:16] Yeah, what info does fsid provide that the /path/to/volume doesn't? [15:55:14] nfsv4 uses the fsuid to identify mounts, and that defaults to the filesystem guid -- which isn't useful when you have the same filesystem /image/ mounted a dozen times at various points in its history. :-) [15:55:27] (I.e.: they all have the same guid) [15:55:58] ok, so then how do I get the fsid for a given volume? [15:56:05] and/or snap [15:56:06] ? [15:57:28] I'm building it arbitrarily 00000000000000000-50380-2013041915 is one, the 50380 is the gid of the project-$foo and the other is the yyyymmddhh of the snapshot (which is actually gettable from the mount point name). the timestamp is 0000000000 for the live volume. [15:57:51] Look at /etc/exports.d/tools.export for an example [15:58:26] Oh, so the for-matti-ng is arbitrary, determined by you? That's easy. [15:59:23] Right, it's just 16 arbitrary bytes. [15:59:35] ok! [16:01:53] petan: where's the source for wmib? [16:09:50] Coren: Do you have some performance tips for Apache/PHP on Labs? [16:10:19] You'll have to define "performance" a bit more. :-) [16:11:08] My apache/php-installations is very slow in my opinion [16:11:30] re [16:11:39] Jan_Luca: Does it do filesystem access a lot? [16:11:40] and maybe do you have some ideas from your tools-setup [16:11:57] Coren: Yes [16:11:58] Jan_Luca: I think I heard about one of your exploits today [16:12:45] Jan_Luca: The problem is probably glusterfs at this time; its performance isn't very good right now. [16:12:46] Oren_Bochman:? [16:13:08] Coren: Ah, should I move the files to /mnt [16:13:22] and only create a backup on /data/project [16:13:24] Jan_Luca: It's worth trying, at any rate. [16:13:40] I'm told you have coded some moodle mediawiki intergataion [16:13:46] Jan_Luca: Things will likely improve once the NFS server is deployed. [16:14:08] Coren: Do you think an opcache would help, too [16:14:54] Oren_Bochman: That's right. At moment I try to get our Moodle installation faster ;-) [16:15:00] Jan_Luca: I doubt it, if it's very IO bound. [16:15:22] I just spent some days installing moodle in labs [16:15:32] it is faster than my local machine [16:15:52] and I have not even tried to make it faster [16:15:58] Coren: Ok, I try to use /mnt instead of gluster [16:16:12] Oren_Bochman: You have a moodle installation, too? [16:16:28] as of this morning - for the israeli chapter [16:16:54] I'm about to announce it [16:16:54] Oren_Bochman: Maybe we could use one together [16:17:06] that would reduce admin work [16:17:53] I can set up an moodle instance inabout 40 minutes ;-) [16:18:14] I almost got it running in puppet too [16:18:40] anyhow I am not opposed to pooling resources/work [16:18:57] but I have to chat with the other project people [16:19:03] Oren_Bochman: I only thought we do not need many moodle instance [16:19:25] well it depends [16:19:53] I started working on a multilingual course [16:20:06] but moodle did not handle this very well. [16:20:43] so I think that are going to have an instance which is for hebrew and possibly one for arabic language [16:21:24] but if it can all be done on one instance - in a statisfactory way I am for it! [16:21:39] Oren_Bochman: Do you speak about the moodle-project? [16:21:46] yes [16:21:48] What is the background of this project [16:22:31] well the WM.IL chapter has arranged a cooperation with our the ministry of education [16:22:55] we are going to teach editing of wikipedia to teachers and students [16:23:16] last month there was such a lesson for 500 students. [16:23:58] Oren_Bochman: I installed Moodle because Wikiversity wants to use it for new ways of courses [16:23:59] I suggested that we make a number (4) courses for teaching editing for different students and for dffernt numbers of lesson [16:24:11] !log tools autofs + gluster = fail [16:24:12] Logged the message, Master [16:24:20] I think your project could be integrated into Wikiversity [16:24:24] !log tools (rebooted -login) [16:24:25] Logged the message, Master [16:24:32] WM.UK also has its moodle project too [16:24:37] I heard about you from them [16:25:05] I also that either wikibooks or wikiuniversity wanted to have moodle [16:25:22] but It was after we got started. [16:25:25] This Moodle stuff is interesting [16:25:30] Coren, other than the literal presense of a snapshot in /time, is there a way to know what snaps are available for a given project? [16:25:35] http://lists.wikimedia.org/pipermail/wikitech-l/2013-April/068498.html [16:25:44] The idea is to use Wikiversity (=wikiuniversity) as a platform to create content for other software [16:26:08] and e.g. include them in Moodle [16:26:26] not a bad idea [16:26:40] I am looking to do a couple of differnt things [16:26:46] so the pros of MediaWiki (open editing) and Moodle (=course management) are combined [16:27:01] andrewbogott: Well, you can look at the actual LVM status, but that's more complicated and what I already do when I maintain the /time mounts. If you want to be extra fancy, you can get the list of possible mounts by enumerating /time, but check that the project directory actually exists /in/ the snapshot before you export it. [16:27:10] this was the reason why I developed my Moodle plugin [16:27:16] I see [16:27:37] andrewbogott: So that you avoid exporting snapshots that predate the project. :-) [16:27:50] Coren: Would it not be easier if they were stored as /time//