[00:18:49] spagewmf: Just one more? [00:19:04] heya, in Nova configure I removed the labs_mediawiki_hostname from i-0000013d (ee-prototype), but it still responds to that even after a puppet run, and it didn't free up a floating instance in our quota. Must I run something else? [00:19:17] andrewbogott: please sir, just one more :) 1 is fine, thanks. [00:20:15] andrewbogott: ^ we're not using ee-prototype so I thought of blanking its hostname and giving it to ee-prototype, but that hasn't worked yet. [00:20:29] sorry, I mean "giving it to shiny new ee-flow instance" [00:20:33] hasn't worked, how so? [00:21:29] ee-prototype still responds at http://ee-prototype.wmflabs.org and our quota still shows 5/5 floating IPs in use [00:22:14] did you release the address? [00:22:55] no, just blanked it in the Nova form "Configure instance i-0000013d (ee-prototype)" [00:23:07] d'oh I see that now, trying it [00:23:50] on Special:NovaAddress, right? [00:24:24] right. And I see you added us a floating IP, so I don't have to do this. [00:24:46] True, but it's still nice to conserve them if you don't need 'em [00:25:07] Anyway, one way or another you should be all set :) [00:31:36] andrewbogott: thanks. I'm confused between the two forms; I associated the IP with ee-flow and aded the hostname in NovaAddress, but it doesn't show as "Public IP" on https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000008a5 [00:32:16] That's a long-standing bug… OpenStack doesn't have a hook for ip changes so that page doesn't get updated properly. [00:32:23] If you reboot the instance it should get updated :) [00:34:46] andrewbogott: yes, piramido shows its public IP in Nova Resource but not toro ("up 128 days" !) [00:34:59] thanks, I think we're set! [05:15:32] [bz] (8RESOLVED - created by: 2Morten Wang, priority: 4Unprioritized - 6major) [Bug 54390] Re-create database views lost in the outage such as arwikiquote_p - https://bugzilla.wikimedia.org/show_bug.cgi?id=54390 [06:29:57] [bz] (8NEW - created by: 2Daniel Kinzler, priority: 4Unprioritized - 6enhancement) [Bug 52693] Allow login using mosh as an alternative to plain ssh on bastion - https://bugzilla.wikimedia.org/show_bug.cgi?id=52693 [06:30:10] !logs [06:30:10] raw text: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ cute html: http://tools.wmflabs.org/wm-bot/logs/index.php?display=%23wikimedia-labs [07:19:09] looking for some help on tools-db: creating function fails due to lack of rights [07:35:26] i'm looking for Coren or any admin of tools-db server [11:58:14] Coren: around? [12:09:22] Ryan_Lane: I miss not having VE in wikitech. [12:09:24] err [12:09:26] you know what I mean [12:09:36] * YuviPanda files bug [12:10:16] [bz] (8NEW - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 54636] wikitech.wikimedia.org needs to have VE enabled - https://bugzilla.wikimedia.org/show_bug.cgi?id=54636 [12:30:41] yurik: Am now. What's up? [12:49:46] Coren: you meant me :P [12:50:14] Coren: can you add role::labsvagrant to be available as a checkbox in all projects / instances? [13:01:37] Yep. [13:02:12] Hm. [13:02:24] Wouldn't it have been saner to call it role::labs::vagrant for consistency? [13:14:49] YuviPanda: Added [13:24:24] Coren: sweet! [13:31:44] Coren: documentation! https://wikitech.wikimedia.org/wiki/Labsvagrant :) [13:31:45] cscott: ^ [13:43:05] cscott: I'll publicize this a bit more if you're able to setup your instance this way without any issues :D [13:43:27] i've added that to my list of things to read/learn. ;) [13:43:46] i spent yesterday lobbying for i18n support, instead of implementing features :( [13:44:14] I partly remember reading that scroll through ;) [13:46:24] yeah, i crashed i18n office hours [13:46:41] at some point i have to implement LanguageConverter for parsoid [13:46:54] aaah [13:46:55] right [13:46:57] ...or whatever replaces LanguageConverter... [13:47:16] the only thing I remember seeing was 'we should merge the dutch and german wikipedias' [13:47:20] or something of that sort [13:47:25] but I was 90% asleep [13:47:26] at that point [13:47:32] so don't quote me on that [13:48:37] i was being provacative. ;) [13:48:46] *provocative [13:48:57] i could see :P [13:49:53] basically there's a broad spectrum of 'language similarity and mutual intelligibility', which doesn't always correlate well with "political divisions powerful enough to name their dialect after themselves" [13:50:02] indeed [13:50:33] it's not always the case, however - many claim you could understand malayalam if you understand tamil, but I can understand Tamil but not malayalam... [13:51:23] sometimes automatic conversion rules or small dictionaries help. [13:51:44] those languages split at least ~1400 years ago tho, so maybe not :P [13:51:54] and have completely different scripts [13:52:00] so possibly not anywhere near like nl and de [13:52:52] brb [13:52:58] urdu and standard hindi are a better example. basically the same language, but different writing systems. [13:53:03] sharply divided politically, though. [13:53:45] wright [13:53:48] right [13:54:09] Languages and their political implications is an area I know enough to know that I know jackshit and usually I keep shut :P [13:54:56] so, getting back to https://wikitech.wikimedia.org/wiki/Labsvagrant -- if i wanted to add a togetherjs role, where do i add it? [13:55:46] cscott: in mediawiki/vagrant.git [13:55:57] cscott: see https://www.mediawiki.org/wiki/Mediawiki-Vagrant [13:56:18] and is that 'safe'? that is, will errors there cause flames and gnashing of teeth from ops? [13:56:19] it has a visualeditor role, so I suspect you'll have a togetherjs role that includes the ve role and adds some more things to it [13:56:24] cscott: not at all. [13:56:32] cscott: you'll notice that it does not live in operations/* ;) [13:56:49] ori, me and bd808 +2 things there all the time, usually [13:56:57] brb [13:57:02] stepping out for an hour [13:57:09] sorry to go so abruptly! [14:49:20] cscott: feel free to ask me questions if you get stuck in adding a role to mediawiki-vagrant. I'm not the puppet-master or anything but I've done several new roles now. Hopefully it will be pretty easy for you. [15:42:15] cscott: back [15:42:17] and +1 to what bd808|AWAY said [16:46:47] [bz] (8UNCONFIRMED - created by: 2metatron, priority: 4Unprioritized - 6normal) [Bug 54653] Cannot create stored function on tools-db - https://bugzilla.wikimedia.org/show_bug.cgi?id=54653 [17:44:32] [bz] (8RESOLVED - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 54636] wikitech.wikimedia.org needs to have VE enabled - https://bugzilla.wikimedia.org/show_bug.cgi?id=54636 [19:18:31] andrewbogott: hm. it seems we have some domain records with arecords [19:18:41] I'm not sure the code can (or ever has been able to) handle that [19:19:10] Ryan_Lane, for example? [19:19:15] beta, wmflabs [19:19:18] I think it's fine [19:19:42] meaning that an IP is assigned to wmflabs.org? [19:19:46] yeah [19:19:50] but I think it's not a problem [19:20:01] we always look up by ip now, right? [19:20:14] For public stuff, yeah. [19:20:35] hm. I wonder if the SOA and everything else needs to be on the same domain [19:20:39] err [19:20:42] on the same entry [19:20:46] * Ryan_Lane checks [19:24:20] ok, so that works fine [19:24:52] for any domain that has an arecord, I'll need to split that out into a new record [19:30:34] andrewbogott: so, for the private entries, the change is that we're renaming the dn to a fqdn, right? [19:30:47] right. [19:31:04] It's a fqdn with the i-000x name rather than the human-readable name… but I think that's what we used before as well. [19:31:54] * Ryan_Lane nods [19:34:58] andrewbogott: we're also using fqdn for domains, too, right? [19:39:59] hm… I don't think I touched the domain code. [19:40:02] * andrewbogott looks [19:42:39] $dn = 'dc=' . $domainname . ',' . $wgOpenStackManagerLDAPInstanceBaseDN; [19:42:47] So, no, currently not the fqdn, just the name. [19:43:04] Hm, seems like that wouldn't work at all. [19:44:26] ok, so I should leave domains alone [19:44:47] Yep [19:46:04] Our current code might cause problems when we migrate… [19:46:18] well, I was going to migrate and deploy [19:48:35] Well, wait. Now that I look back at the gui, it handles pmtpa vs. eqiad explicitly. So having a single domain entry for both is probably correct. [19:49:17] oh, now I'm getting even more confused, because those domains don't go under .pmtpa they go under wmflabs [19:49:32] * andrewbogott tries to create foo.foo.foo.wmflabs.org [19:50:33] sure enough, that does not work. [19:50:53] So, irrelevant to our migration, but the current code doesn't allow reuse of the same subdomain under different parent domains. [19:51:03] * andrewbogott logs a bug [19:51:59] heh [19:52:56] this is one of the reasons I can't wait to switch to designate :) [19:53:23] it's going to take me ages to make all these dns changes [19:53:33] [bz] (8NEW - created by: 2Andrew Bogott, priority: 4Unprioritized - 6normal) [Bug 54664] cannot create foo.foo.wmflabs.org domain - https://bugzilla.wikimedia.org/show_bug.cgi?id=54664 [19:59:46] Rebuilding a 30T raid6 == really "fast". Yeay. [20:15:51] I should have wrote a script for this :( [20:16:05] I guess I still could [20:16:17] * Ryan_Lane does [20:21:18] heh. well, that was quick [20:54:10] andrewbogott: ok, so, I have a couple files on virt0 I'd like you to check out [20:54:32] ok [20:55:15] /root/public-hosts.ldif [20:55:22] /root/private-hosts.ldif [20:56:20] I can't do modrdn for the public hosts, since we're changing their entire tree [20:56:29] so I need to create an entry, then delete an entry [20:56:43] for the private hosts I can do a modrdn [20:56:54] I did a test with a couple of entries to make sure this would work [20:57:14] * andrewbogott looks [20:57:59] dnschange/* ? [20:59:05] /*? [20:59:15] oh [20:59:17] whoops [20:59:17] yeah [20:59:57] ok, I merged all your changes [21:00:08] now it's just making the dns changes, if good, and then deploy [21:00:34] oh, I need to remove the sOA attributes [21:00:49] since those aren't domain records [21:01:31] newrdn: dc=i-0000007a.pmtpa.wmflabs, not newrdn: dc=i-0000007a.pmtpa.wmflabs,ou=hosts,dc=wikimedia,dc=org? [21:01:42] Or is that implicit somehow? [21:07:47] andrewbogott: it's implicit [21:07:52] ok. [21:07:58] it renames the entry in the same base [21:08:07] In that case the first few records of each ldif look ok to me. [21:08:22] The thing you're saying about sOA records… does that mean the code is still wrong? I think it adds sOA for each host. [21:08:25] there's a way to rename between trees, too, but it's a pain and is relatively non-standard [21:08:45] andrewbogott: there's two entries that are both domain and arecords [21:09:07] Oh, looking at the code, it doesn't do what I thought. [21:09:08] so only two are wrong there [21:09:09] So, nevermind! [21:09:11] Yep. [21:10:24] labs-dev unresponsive [21:10:31] oh. hm. [21:10:34] at least on ssh [21:10:57] Danny_B: you mean tools-dev? [21:11:23] andrewbogott: hm. should dc=i-0000007a,dc=pmtpa,ou=hosts,dc=wikimedia,dc=org be dc=i-0000007a,ou=hosts,dc=wikimedia,dc=org ? [21:11:45] err [21:12:03] should it be dc=i-0000007a.pmtpa.wmflabs,ou=hosts,dc=wikimedia,dc=org ? [21:12:53] um… yes, it should! oops. [21:13:09] ok, I need to generate those like the public entries too, then [21:17:37] andrewbogott: ok, now does private-hosts.ldif look ok? [21:17:37] I can exercise the new code on nova-precise2 and do an ldap query if it's useful to have examples [21:18:09] that could be useful, yeah [21:18:39] Ryan_Lane: yup, sorry [21:20:38] Coren: ^^ [21:20:56] I'd take a look, but I'm in the middle of a relatively complex change [21:21:06] * Coren reads backlog. [21:22:17] Danny_B: Ah. I've seen it was very loaded earlier. Lemme see if I can see what's up with it. [21:22:20] Well, dammit, I have have uncovered a new bug in my code. Stand by... [21:22:39] my ssh is still unrensponsive [21:23:31] It, seemingly, went "boom" [21:24:27] * Coren kicks it. [21:25:25] Somebody ate all the memory ~40m ago; and it hasn't recovered. [21:27:42] Danny_B: It lives! [21:28:12] not my ssh yet [21:28:47] Any sessiou you might have had won't have survived the reboot; but I can ssh to it okay. [21:29:27] * Coren has definitely been too stingy with the resources for the bastions. [21:47:02] andrewbogott: what's the bug? anything I can do to help? [21:47:13] I have a fix, just testing and cleaning up. [21:47:18] cool [22:01:47] Ryan_Lane, first fix is https://gerrit.wikimedia.org/r/#/c/86182/, another coming up... [22:13:44] grrrr [22:14:12] Ryan_Lane, does the fancy JS interface mean that specialNovaInstance::tryDeleteSubmit is no longer the code for deleting an instance? [22:15:05] Or is it cached somehow? I'm working on that bit of code and I can't ellicit a response [22:16:02] oh [22:16:07] it's not. it's in the API now [22:16:21] though it's also there, if JS isn't used [22:17:04] So where should I look for the active code? [22:17:44] api directory [22:17:59] ApiNovaInstance.php [22:18:10] case 'delete' [22:18:12] ok [22:18:28] it just calls: $host->deleteHost(); [22:22:15] I wonder if this was always broken… when looking at the product ldap did you find a lot of host records for instances that don't exist anymore? [22:28:26] I think there's a few in prod like that [22:28:33] but in general it should be relatively clean [22:28:49] in labs it's different because we just imported a bunch of crap from production for testing [22:29:26] Right now when I delete an instance the host persists forever. [22:34:39] [bz] (8NEW - created by: 2Chris McMahon, priority: 4Unprioritized - 6major) [Bug 54671] VisualEditor opt-in preference disappears - https://bugzilla.wikimedia.org/show_bug.cgi?id=54671 [22:38:58] [bz] (8NEW - created by: 2Jack Phoenix, priority: 4Unprioritized - 6normal) [Bug 53635] div.navbar-inner obscures content on Internet Explorer 10 - https://bugzilla.wikimedia.org/show_bug.cgi?id=53635 [22:40:39] andrewbogott: oh? [22:40:43] andrewbogott: why's that? [22:41:03] loadHost() is failing in the instance. Still tracking it down... [22:46:11] OK, I may be making some basic php error. Look at getPrivateHost(). Is the logic for whether or not to return null there sound? [22:46:19] one sec [22:50:52] andrewbogott: is nova/OpenStackNovaInstance.php calling it? [22:50:53] in this case? [22:51:04] yeah. [22:51:25] OK, so, I've narrowed this down further… this line is returning nothing. [22:51:26] $result = LdapAuthenticationPlugin::ldap_search( $wgAuth->ldapconn, $wgOpenStackManagerLDAPInstanceBaseDN, '(dc=' . $fqdn . '))' ); [22:51:39] Where $fqdn is i-00000052.pmtpa.wmflabs [22:52:03] When I look in ldap I see a record: dn: dc=i-00000052.pmtpa.wmflabs,ou=hosts,dc=wikimedia,dc=org [22:52:08] So, is my ldap_search string wrong? [22:52:18] (I can't imagine that this wouldn't be failing all over the place in that case...) [22:52:49] let's see what your query looks like... [22:52:53] this is wikitech-test? [22:52:58] yep. [22:54:18] [26/Sep/2013:21:46:24 +0000] SEARCH REQ conn=818278 op=6 msgID=10 base="ou=hosts,dc=wikimedia,dc=org" scope=wholeSubtree filter="(associateddomain=i-00000052.*)" attrs="ALL" [22:54:22] that's the only query I see [22:55:17] for that instance id anyway [22:58:17] which directory is this deployed in? [22:58:50] I don't see it in w, w2 or w3 [22:59:08] * andrewbogott curses [22:59:11] mismatched parens [22:59:21] which must've been producing no error, just an empty search [23:02:56] https://gerrit.wikimedia.org/r/#/c/86197/ [23:05:06] anywayyyyyyyyy, the file that I was trying to make for you is now /root/testsample.ldif on virt0 [23:06:48] heh [23:06:49] thanks [23:08:13] ok, cool [23:08:16] and private-hosts.ldif looks right to me, but clearly my eyeballs are not to be trusted [23:08:17] this matches what I have [23:11:18] ok. going to deploy and make the dns changes [23:11:31] * andrewbogott crosses fingers and toes [23:11:44] :) [23:12:25] bleh. I forgot to add a changetype to the entries [23:12:33] that would have ended poorly [23:17:20] private hosts done [23:17:55] deployed the change [23:22:22] public change done too [23:22:26] I'm going to purge memcache [23:22:51] looks like it's working [23:23:49] yep [23:23:54] public dns is good [23:24:25] checking instance creation [23:24:57] getting a blank page on instance creation [23:25:12] PHP Fatal error: Call to a member function getFullyQualifiedDomainName() on a non-object in /srv/org/wikimedia/controller/wikis/slot1/extensions/OpenStackManager/nova/OpenStackNovaHost.php on line 175 [23:26:49] andrewbogott: ^^ [23:27:01] dammit! [23:27:09] Um… did you already merge those last two patches on virt0? [23:27:31] yep [23:28:59] heh. I need to turn whitespace checking back on, there's some whitespace issues in the code :) [23:29:05] I missed some formatting errors, too :D [23:31:25] Hm, can't reproduce the error in testing [23:31:55] purge memcache [23:34:21] ah. crap [23:34:25] puppet is broken now :D [23:34:47] Nah, all works great on wikitech-test :( [23:34:57] Wait, puppet now? [23:34:58] hm [23:34:59] Related? [23:35:04] yep [23:35:09] how so/ [23:35:10] ? [23:35:11] certname = i-0000019b.pmtpa.wmflabs.pmtpa.wmflabs [23:35:16] eh [23:35:18] err [23:35:19] heh [23:35:21] hm [23:35:26] I'm on that [23:35:28] ok [23:38:15] newly created instances look right in ldap to you? [23:38:19] (they do, to me) [23:40:01] are they being added? [23:40:39] oh, they are [23:40:56] I wonder why that error is being thrown [23:41:08] of course, no arecord is being added for it [23:41:25] and the entry isn't being deleted when the instance is deleted [23:42:36] heh. crap [23:42:40] salt is broken in the same way not [23:42:41] *now [23:42:58] that's going to make things.... difficult [23:43:12] I can accept the certs for puppet and salt [23:43:20] then run puppet and delete the new certs [23:44:40] The no arecord thing is expected, though, right? Because they don't have an IP at creation time? [23:45:34] a job should fix that, though [23:45:43] oh, I wonder if the job needs to be fixed in some way? [23:48:22] did beta labs just disappear completely? no DNS? http://en.wikipedia.beta.wmflabs.org [23:48:32] crap [23:48:39] :( [23:48:59] chrismcmahon: as mentioned in labs-l, we're doing massive DNS changes today [23:49:15] seems to have been very recent, but beta looks like toast right now [23:49:35] like in the last few minutes recent [23:49:46] again. see my very last comment [23:50:18] hm. the * entry seems to be completely missing [23:50:30] I wonder if it failed to add for some reason [23:52:36] seems to be back now [23:53:13] yeah, but anyone with a negative cache is going to be screwed for a bit [23:53:20] I just added the * entry back in [23:53:43] let me see if any other entries are missing for beta [23:53:51] beta was one of the special entries in ldap [23:54:32] I think just that was missing [23:54:45] ok, now to fix puppet