[00:08:00] hexmode: You can't take things word-for-word from wikitechy, we're not there to being that close a copy [00:08:20] /usr/local/apache/common/bin/updateinterwiki [00:08:27] johnduhart: I see that and found that script [00:08:34] but even after running [00:08:40] still no fix :( [00:08:56] What are you tring to fix? [00:09:08] labs [00:09:14] but looking at the script [00:09:20] it might not be in there [00:10:38] I don't get it, what's wrong? are you trying to add a prefix for labs.wm wiki? [00:10:44] andrewbogott: it would also be interesting to store ssh host keys in LDAP, when creating the entry [00:10:55] I don't think that would be terribly easy, though [00:10:59] johnduhart: the bugzilla: prefix isn't working on labs [00:11:12] ah [00:11:15] the interwiki table *was* wrong [00:11:16] since the host key is created after boot, and nova doesn't have access to it [00:11:19] and I fixed it [00:11:24] hexmode: How? [00:11:25] but still, nothing [00:11:27] Putting extra stuff in ldap sort of presumes that any given dns driver will support storing arbitrary extra info. [00:11:36] We could store it in the nova db though. [00:11:48] johnduhart: using sql.php and "UPDATE..." [00:11:50] well, the ssh host key in LDAP is just another kind of DNS record [00:11:58] http://www.ietf.org/rfc/rfc4255.txt [00:12:05] an SSHFP record [00:12:27] hexmode: Well that won't do any good, the table's contents are dropped when the update script is ra [00:12:28] n [00:12:29] so it would work with any kind of DNS server [00:12:35] heh [00:12:45] johnduhart: so what do I need to do? [00:12:49] * hexmode is lost [00:13:06] Not sure, let me look at the table quickly [00:13:27] johnduhart: the wikifarm is using cdb files [00:13:30] hm. I wonder what attribute would store that [00:13:36] at least that is how it is set up [00:13:53] hexmode: That's a cache, it's still stored in the database [00:14:01] right [00:14:32] johnduhart: looking at the script [00:14:39] Ryan_Lane: OK, that seems possible then, in a later version maybe. [00:14:43] yeah [00:14:44] it looks like it copies the interwiki table from enwiki [00:14:50] definitely not this iteration :) [00:14:50] and then updates sql [00:14:56] then the cdb [00:14:57] hexmode: Not percisely [00:15:05] I found that issue, hold on [00:15:08] I'm trying to think of ways to handle our ssh host key issue [00:15:30] right now when an instance is deleted and recreated, everyone's trust list gets screwed up [00:19:33] This is why I didn't want to create a million wikis [00:19:40] Because it takes forever to update [00:19:54] hexmode: make sure you use the protocol-relative cdbs! [00:20:04] or, well, the version of the script that creates them [00:20:19] Ryan_Lane: since I'm not doing it... [00:20:20] Ryan_Lane: We're http: only here anyway [00:20:39] johnduhart: def could use the ability to update only one [00:20:42] it's better to make it look like production [00:20:45] * hexmode sighs [00:20:55] it's like one flag in the script to do it [00:21:12] I removed the flag when I adopted it for here [00:21:18] o.O [00:21:32] * hexmode has been having a ton of fun with labs and deploy, but feels like his bugs are lonely [00:21:56] we run protocol-relative in production. if we don't in labs, we may miss a bug [00:22:02] also, we should likely run https [00:22:03] johnduhart: yeah protocol-relative is relatively un-tested. It'd be better to have it [00:22:18] we *can* run https, with the right certificate hack [00:22:25] johnduhart: but this is not something that blocks us [00:22:50] Put it on the queue, there's more important stuff to do than that right now [00:23:14] I'd like to fix some of the issues that have popped up with MW and then get a round TUIT later [00:23:18] hexmode: relatively untested? it's being tested live! :D [00:23:42] Ryan_Lane: relative as in it is < 1year old [00:23:46] heh [00:23:52] I know. I'm messing with you [00:24:59] what the fuck [00:25:05] ? [00:25:09] why the fuck is /mnt/export mounted on /tmp [00:25:18] that's a good question [00:25:36] * johnduhart bangs head on desk [00:26:38] ugh [00:26:49] Need to stop apache so I can unmount this. [00:35:22] PROBLEM HTTP is now: CRITICAL on deployment-web deployment-web output: Connection refused [00:36:18] thanks SO http://stackoverflow.com/questions/40317/force-unmount-of-nfs-mounted-directory [00:38:42] !log deployment-prep Unmounted /mnt/export from /tmp on -web [00:38:43] Logged the message, Master [00:40:52] did maxsem set up abusefilter? [00:40:58] * hexmode goes to look [00:41:40] oh, still down? [00:41:59] Yup, hold on [00:42:05] k [00:42:40] OrenBochman: where do we stand with search? [00:45:43] * hexmode resorts to email [00:48:52] lol snowolf [00:50:22] RECOVERY HTTP is now: OK on deployment-web deployment-web output: HTTP OK: HTTP/1.1 302 Found - 565 bytes in 0.004 second response time [00:52:07] petan: .... [01:08:20] hexmode: Bugzilla links should be good now [01:08:33] Needed to restart memcached for it to take effect [01:10:40] johnduhart: ugh [01:10:51] tyvm, though [01:14:26] andrewbogott: I know you're going to kill me, but can you switch to using novaadmin2? [01:14:43] pdns is configured to use novaadmin with the private password [01:14:44] in puppet [01:14:49] switch from nova-dev2? Or switch the ldap url? [01:14:51] same password [01:15:03] different nova admin user [01:15:08] oh, sure. [01:15:08] hexmode: we have a couple of issues [01:15:09] oh. I need to do some other changes too [01:15:12] one sec [01:15:20] johnduhart: ?? [01:15:25] ? [01:15:36] johnduhart: oh, I meant OrenBochman [01:15:40] OrenBochman: ?? [01:15:48] andrewbogott: ok. done making the changes [01:15:51] hexmode: I'm not sure how big we should make the new instances [01:15:58] uid=novaadmin2,ou=people,dc=wikimedia,dc=org <-? [01:16:02] yep [01:16:06] ok [01:16:08] same privileges, different password [01:16:16] OrenBochman: how big do you think we could start with? [01:16:20] err [01:16:22] sorry [01:16:28] also we need to automate pushing all the global config to the indexer [01:16:29] OrenBochman: and how big will Ryan_Lane let you go ;) [01:16:29] andrewbogott: it has the new password you wanted [01:16:41] the novaadmin user has the original password :) [01:16:55] how big do you need it, and why does it need to be huge? [01:17:06] OrenBochman: global config... can you just get what you need via nfs? [01:17:09] Ryan_Lane: "...cannot be added due to insufficient access rights" [01:17:15] OrenBochman: or is this OAI [01:17:16] ? [01:17:17] hm [01:18:14] andrewbogott: it'll work now [01:19:18] sec [01:19:25] k [01:19:41] andrewbogott: ok. dns searches are working [01:19:47] dig @10.4.0.61 i-0000002d.pmtpa.wmflabs [01:19:52] I'm still getting insufficient access [01:19:57] crap. really? [01:20:00] OrenBochman: should I see if Brion is around or do you need someone else? [01:20:28] RAWR [01:20:37] OrenBochman should talk to me about wiki configuration stuff [01:20:51] nfs may be ok [01:21:12] andrewbogott: ok. I got it for sure this time :) [01:21:27] works! [01:21:31] * Ryan_Lane remembers that you can't just copy and paste somethign and expect it to work [01:21:33] also I've been told by petan that OAI is installed and working on the tarkge machines [01:22:38] OrenBochman: have you tested it? also what is tarkge? [01:23:35] johnduhart: where is the updateinterwiki script run from? [01:23:40] which machine? [01:24:09] -web or -test, but peter wants maint stuff run on test [01:24:16] hexmode: I don't know how to test it [01:24:20] Doesn't matter that much, both work [01:24:25] hexmode: or how to install it [01:24:59] OrenBochman: k, so what else can you do? or is this more in petan's realm now? [01:25:13] I only know from the docs that the search side will work as long as the OAI section of the globalconfig is good [01:25:20] k [01:25:39] I *think* Brion knows about that bit and will check with him [01:25:43] see if he can help [01:26:39] that would be great - especialy if the docs were updated - they are in a disgracefull state [01:26:53] I have a couple of issues [01:27:11] k, so I'll get that from Brion... next issue? [01:27:22] the docs state that you have to make the global config by hand for this type of setup [01:27:59] I can try but I've only done one installation of search so far [01:28:00] what about oai? [01:28:24] OrenBochman: there he is, ask him! :) [01:28:30] get him! [01:28:52] we'd like to know how to test if it is working [01:29:14] Ryan_Lane: I'm going to be installing Gluster tomorrow at school :O [01:29:27] johnduhart: it's incredibly easy [01:29:41] Ryan_Lane: I know, I was shocked when I read the doc [01:29:52] brion: we'd like to know how to test if it is working [01:29:55] also I'd like to know how to install it [01:30:08] andrewbogott: http://mailman.powerdns.com/pipermail/pdns-users/2011-March/007547.html [01:30:10] \o/ [01:30:15] damn it [01:30:16] heh [01:30:26] brion: also I'd like to know how to install it [01:30:54] brion: the extention docs are out of sync [01:31:00] dang. [01:31:13] also, SSHFP records aren't supported by the LDAP backend [01:31:14] also they are rather vague [01:31:14] OrenBochman, pop over to say https://en.wikipedia.org/wiki/Special:OAIRepository [01:31:19] it'll give you an HTTP auth prompt [01:31:21] win [01:31:31] since i don't think anybody cares these days i'll just give you the testing credentials ;) [01:31:35] user 'testing' pass 'mctest' [01:31:59] see http://www.openarchives.org/OAI/openarchivesprotocol.html for general protocol documentation [01:32:15] to install locally.... in theory: [01:32:24] the special page asks for a password [01:32:26] make sure you've got OAI extension dir in place [01:32:30] ^ see the user/pass above [01:32:55] and do the usual require "$IP/extensions/OAI/OAIRepo.php"; [01:32:58] oh well. we'll stick with the LDAP backend for now [01:33:00] you'll only need the repository half [01:33:15] later we can write a file or database based backend [01:33:19] run maintenance/updates.php to make sure it installs its tables... which i think should work [01:34:05] much, much later :) [01:34:15] as pages get edited/created/deleted, it'll internally record things into its table [01:34:24] ok [01:34:27] and those records can get read out through the Special:OAIRepository iface [01:34:43] does it store the changes or just which pages are dirty ? [01:35:17] iirc it records the page id (?), possibly a rev id, and a created/edited/deleted state flag [01:35:39] then the interface slurps out current page content [01:35:44] at request time [01:36:01] it's meant to give you current versions of stuff, rather than to show you every individual change [01:36:21] (eg, potentially multiple changes since your last query will be "rolled up" into one, and you just download the entire page text as of the last change) [01:36:44] or if it's deleted, you get a marker indicating the page was deleted [01:36:46] ok [01:37:17] the docs you pointed me too is interesting but too much information right now [01:37:18] it's relatively straightforward, but doesn't always map to what folks want :) for search index updates it's good enough... as long as you're working with source [01:37:35] all you probably need is the 'ListRecords' verb [01:37:46] ok [01:39:05] metadataPrefix too [01:41:21] ok great I can test it! [01:41:39] yay :D [01:41:44] brion:thanks [01:41:59] np [01:42:48] Ryan_Lane: any progress about console access to the production search servers ? [01:43:19] OrenBochman: I asked. consensus was that we didn't want to give out new shell access [01:43:36] we'll likely take the current config and add it to the git repo [01:43:45] even if it isn't actually used by puppet [01:43:52] and walk you through things [01:44:15] ok [01:45:30] Ryan_Lane: there are scripts running things there which are undocumented [01:45:36] yes [01:45:37] I know [01:46:28] so why not zip them up to somewhere [01:47:04] and also provide a directory layout of how things are orgenised there [01:47:22] and how are nfs mappings are set up [01:47:26] we need to sanitize that stuff, likely [01:48:18] I don't realy care about that stuff so much since I'm going for a completely different architecture [01:48:35] well, for what's needed in deployment-prep we need it [01:49:49] I agree - but perhaps peterb should be the dude on point - I'm just guessing what should be happening - he can go back and forth [01:50:05] and check what is going on [01:51:11] I sent rainman a bunch of questions about to over view of the production but he has not gotten back to me so far... [01:51:25] Ryan_Lane: So, what is the default backend for pdns if not ldap? [01:53:04] I help out as much as I can. [02:01:36] I was hoping that the search-test would allow testing - but the unit test don't work on that machine either [02:03:34] johnduhart: around? I thought I had enough info from you to update interwiki, but apparently not [02:04:04] I'm going to have far less time from Monday - I was hoping to run test against the code I wrote in my vacation. [02:05:19] andrewbogott: hm. the pdns backend. I dunno what it looks like, though [02:05:26] in production we use the bind backend [02:07:40] OrenBochman: Is there anything I can do to help you get your code tested? Chase down rainman or something? [02:08:50] I'm trying to figure out what's wrong - most of the test run on my windows machine both before and after my chenges [02:09:56] I'm not very good with subversion and I did not want to mess things up with untested code. [02:22:56] hexmode: what's up? [02:24:11] Ryan_Lane: Could I get added to the testlabs project? [02:24:20] johnduhart: I was trying to update the mw prefix and couldn't ... thought I had done everything -- even restarted memcached -- but no [02:24:26] there's really nothing going on in there, why do you need access? [02:24:56] nothing was finished being configured [02:24:57] Ryan_Lane: After this I wanted to work on that, or is it ops only [02:25:06] ahhhhh. ok [02:25:20] hexmode: Update the mw prefix? [02:25:23] I was thinking we could likely just treat deployment-prep as if it was that [02:25:37] johnduhart: interwiki prefix for mw [02:25:43] or is there a reason not to do so? [02:26:06] Ryan_Lane: deployment-prep is not puppetized. At all ;) [02:26:17] it should be ;) [02:26:25] that's the next logical step [02:26:25] Think of it as a first pass to getting testlabs working [02:26:37] ah. ok [02:26:49] I need to move the NFS home directories out of that project first [02:27:26] and I was thinking of moving them to gluster. [02:27:41] Ryan_Lane: Also question, is the whole "developers can create wikis to test stuff yadda yadda" seperate from testlabs? [02:27:42] so, we may want to wait until gluster is done. [02:27:47] Ryan_Lane: sure [02:27:58] no. the idea is that would be testlabs [02:28:14] there shouldn't be a need to have root on the instances to do so [02:28:59] to create new wikis, that is [02:31:32] Alright, so testlabs is to allow developers to create instances of MW to test and play with with things, while having conditions similar to production. Is there a focus anywhere of creating a mirror of production, like testwiki or prototype on a large scale? [02:32:03] well, that's also testlabs [02:32:10] there's no reason it can't be both :) [02:33:27] ok. internets are likely going away for me in a minute [02:34:22] the difficult part for testlabs will be the proxy configuration [02:34:38] or the apache configuration, I guess [02:34:49] the proxy config will likely be fairly straightforward [02:35:05] ugh desktop crashed [02:35:09] :D [02:36:00] The thing I was concerned with about both coexisting is the infastructure for the WMF mirror config and the developer wiki creation will be vastly different [02:36:18] why would it be different? [02:36:57] the only major difference would be that the apaches would have multiple installs [02:37:04] of mediawiki [02:37:48] we could have a deployment host [02:37:56] Ryan_Lane: One is several hundred wikis sharing the same codebase and using one configuration chain, the other is several different versions of MW, different installs, each needing a seperate config [02:38:23] yeah. I don't think that's a problem [02:38:54] each differently installed wiki can have an apache file dropped in place [02:39:05] pointing at the install location for the wiki [04:59:46] looks like hexmode decided to release the flood :) [05:00:07] ryan's internet sucks [05:00:11] 12 05:00:07 < jeremyb> ryan's internet sucks [05:00:16] jeremyb: wikitech-l isn't exactly a flood [05:00:19] welcome :) [05:00:35] I'm on a mifi, and the battery died [05:00:37] hexmode: well not slashdot :P [05:00:54] jeremyb: oo! [05:01:06] I could post this on reddit! [05:01:11] meh [05:06:50] bye again [05:24:32] huh? [05:25:28] wondering the same [05:28:00] hexmode: Wait did we ever fix the mw interwiki? [05:34:47] this deployment cluster looks really great guys [05:34:49] good job [05:34:55] Thanks [05:35:19] Config for most of it is here https://github.com/johnduhart/deploymentprep-conf [05:35:58] in github? :D [05:36:27] Yeah, there's a git repo with all the config in it and I push it when I can :) [05:36:33] heh [05:36:51] we could have made a gerrit repo for it too [05:37:15] I guess I need to make a gerrit manager [05:37:21] that'll let people create their own repos [05:37:30] I guess so [05:37:39] * johnduhart works on Math extension [05:37:42] heh [05:37:49] have fun with that one [05:37:51] it's a PITA [05:38:08] I'm sure we have the requirements puppetized [05:38:36] I'm pretty sure I have them all installed something probably just has the wrong permissions [06:05:01] Ryan_Lane: chown: changing ownership of `upload': Remote I/O error [06:05:03] Ideas? [06:05:11] where at? [06:05:20] /mnt/upload on -web [06:05:27] It's an NFS mount share [06:05:30] gimme a sec [06:05:36] sure, thanks [06:06:04] That's the math problem, the rename() returns false because it can't change owner, even though the file made it across [06:06:13] ah [06:06:49] whats the specific file? [06:06:56] and which user is trying to write it? [06:06:58] wow [06:07:03] What? [06:07:07] the ownership is *really* fucked up [06:07:11] yup [06:07:12] did someone use NFS4? [06:07:19] uh [06:07:24] what's the nfs server? [06:07:24] maybe... [06:07:32] deployment-nfs-memc [06:07:45] nfs *and* memcache? :D [06:07:53] This was a "there's no one else around and I want to get this done, it can get fixed later" [06:08:01] heh [06:08:04] Ryan_Lane: heh not much choice :D [06:08:07] my* [06:08:11] thats' funny [06:08:29] normally I'd say, sure stick memcache most places [06:08:30] The permissions are fine on -nfs-memc [06:08:56] ,fsid=0 [06:09:09] not needed for 3 [06:09:12] AGAIN I DONT KNOW WHAT IM DOING [06:09:16] :) [06:09:20] :D [06:09:21] I know, I'm showing you [06:09:29] thanks [06:09:41] I don't think insecure is needed either, but let me make sure [06:10:11] yeah, that can be removed too [06:10:24] usually you'd want this: rw,no_root_squash,no_subtree_check [06:10:37] I think sync is the default [06:11:28] ah. maybe sync is *not* default [06:12:37] seems sync is a sane option there [06:14:16] no_subtree_check isn't needed here [06:14:27] this is when a file is renamed when a client has it open [06:14:33] no worries about that here [06:14:41] ah [06:14:42] so, I'm gonna remove fsid=0 [06:14:49] cool [06:14:50] that's an NFS4 option [06:15:10] lemme see about insecure [06:16:01] secure/insecure are which ports are used [06:16:11] secure says ports below 1024 will be used [06:16:19] no need for insecure here either [06:16:32] I'll leave sync [06:16:40] it helps ensure there isn't data loss [06:16:48] it's going to hurt performance, though [06:17:05] ugh [06:17:07] right [06:17:16] exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "10.4.0.0/24:/mnt/export/". [06:17:16] Assuming default behaviour ('no_subtree_check'). [06:17:16] NOTE: this default has changed since nfs-utils version 1.0.x [06:17:23] adding no_subtree_check back on [06:17:54] ok. now for the client [06:18:01] ok [06:18:08] how many clients are mounting it? [06:18:16] just the one? [06:18:22] just -web [06:18:25] ok [06:18:53] how is it being mounted? [06:19:10] I ran mount [06:19:16] ah. by hand [06:19:22] * johnduhart nods [06:19:25] deployment-nfs-memc:/ on /mnt/upload type nfs4 (rw,clientaddr=10.4.0.37,addr=10.4.0.58) [06:19:27] too cool for fstab [06:19:31] heh [06:19:41] so, notice it's running nfs4 [06:19:46] security works different with nfs4 [06:20:06] oh? [06:20:11] eventhough it's using auth=sys, the domain doesn't match [06:20:27] auth=sys means the server hands control over to the client [06:20:35] nfs stands for no fucking security [06:20:41] lmao [06:20:56] if the server trusts the client's IP, the client is the one who had full control [06:21:19] but NFS4 also does server mapping of users and groups by domain [06:21:22] nfs3 does not [06:23:09] so. let's remount this [06:23:18] mount it as nfs rather than as nfs4 [06:23:20] Sure [06:23:22] let's also make fstab mounts [06:23:26] heh [06:23:34] Should I stop apache for a sec? [06:23:37] there's a way to make this work for nfs4, but I can't find the docs [06:23:46]