[00:58:08] petan: You around? [04:02:51] hi. re. petan pong [04:03:15] He's probably sleeping, it's like 5/6 am [04:03:53] yep, i know, Im just jetlagged. He'll see it later [04:04:21] I wish I was jetlagged, just can't sleep :P [04:04:34] any open requests for accounts or something? [04:04:50] There was a new project request I think [04:04:58] !requests [04:04:58] this is a backlog of all requests needed to be done by ops https://labsconsole.wikimedia.org/wiki/Requests [04:04:59] :) [04:05:02] Not been paying attention, rather busy choking to death. [04:05:10] wow, it actually works. that was a wild guess [04:05:16] :) [04:06:22] Damianz: choking to death sure doesnt sound good :( [04:06:29] OTRS is the helpdesk the weird people that go bitch at you for using freelylicensed images that don't happen to be cc-by-sa licenses isn't it? [04:07:40] heh, afaik that definition might fit pretty good [04:07:50] but it is more general "handle queries, complaints, and comments from the public" [04:08:13] Ah [04:08:25] http://meta.wikimedia.org/wiki/OTRS [04:08:52] I hate the whole cc-by-sa license thing :( Free and open are either free and open or their not... bloody legal people, let's make everything WTFPL! [04:11:56] some people set up their own mediawikis and use cc-by-nc-sa, because they don't want people to use their content commercially, then they import stuff from Wikipedia, to then realize they are not allowed to, they are less free, adding an "nc" to it. [04:13:07] Trying to import Wikipedia stuff is a task and a half, IIRC wikipedia have all sorts of weird syntax hacks in place for templates etc. [04:14:00] well, there is Special:Import , Special:Export [04:14:20] Hmm true :P [04:16:20] project can't be named "OTRS", just "otrs", a-z only, no A-Z [04:17:25] Upercase is pretty much hated on unix based systems anyway :P [04:18:54] hrmm, one could argue just cause Windows users don't realize upper and lowercase are not the same character [04:18:58] but yea [04:20:09] One thing I don't like so much about MW is the forced case on the username.... makes sense for sigs/links etc but for the login form!? [04:22:01] agreed, especially because you don't need to have it active in all namespaces, sometimes i found it very confusing if it is active in Main: and User: but NOT in Mediawiki: with the template names [04:22:13] but i think there has been endless discussion on that before somewhere;) [04:22:33] it's actually a configurable setting in MW though [04:23:14] lol [04:23:20] MW has some insane configuration options. [04:24:23] "To fix this problem, place the {{lowercase}} tag at the top of the article page (and optionally at the top of its discussion page). This will cause the article title to be displayed with the initial letter in lower case, as at eBay. Note that it does not fix every occurrence, like the history, edit, or log pages. [04:24:31] :p [04:25:28] I hate to think what that does code wise if it dosn't fix everything I assume it's not db wise that's doing it... though MW's syntax parser is an ungly thing to try and replicate. [04:25:29] http://www.mediawiki.org/wiki/Manual:$wgCapitalLinks [04:26:00] <-- and the talk page [04:27:08] full ack @ syntax parser . really big deal to rewrite that [04:27:09] 'So how can this forcing of capital letters be removed completely? I am quite anal and really do not like this' LOL [04:27:16] haha [04:28:17] "You can force this only by hacking the source code. This will create a whole host of other issues and it NOT recommended. You will find that you will be spending a great deal of time hacking code to fix problems." [04:28:36] brb, get some coffee [04:28:53] Yeah, doing that isn't fun.... it's bad enough when extensions stop being maintained and you get stuck at 1.6 with a rewrite something totally not compatible or drop it issue. [04:34:01] like the sorry state of LiqudThreads [04:44:38] yes, it sucked when the "Task extension" didnt work anymore for me [04:44:59] Try not to touch LiquidThreads but there again my wiki usage isn't very wiki like, more kb like. [05:00:25] 03/29/2012 - 05:00:25 - Creating a home directory for jeremyb at /export/home/otrs/jeremyb [05:01:26] 03/29/2012 - 05:01:26 - Updating keys for jeremyb [05:01:52] !log otrs - tried to create the project because it was listed in !requests and it failed .. because it already existed ;p, so just added JeremyB as a member [05:03:28] lol [05:12:48] !log hello logbot [08:45:23] mutante: ping [08:45:33] pong [08:45:37] I forgot what I wanted [08:45:41] but hi [08:46:30] methecooldude: yes I am here [08:47:10] mutante: can we try the other way to check puppet [08:47:14] the one in gerrit [08:47:22] I will check if it works and then I ping you again [08:48:25] !whatIwant is I want to ask mutante to merge my fix and I want to fix deployment site [08:48:25] Key was added! [08:48:30] jic [08:48:32] petan: i just found !requests earlier, it's cool!, i just typed it to suggest to create it, and it actually worked :) [08:48:44] really? I am sure it wasn' [08:48:46] t me [08:48:49] petan: just that we also need a way to strike/mark as done requests [08:48:52] who created it [08:49:02] I never saw that page [08:49:03] :D [08:49:04] it asked to create a project that already existed [08:49:11] oh, heh, ok [08:49:21] or I think so [08:49:30] it might be so long ago [09:26:58] andrewbogott: hi, the recovery of sql database is probably going to be very hard, since the oldest backup we have, seems to be corrupt as well [09:27:20] I have no idea how is that possible... [09:28:20] Can't you just re-import the dumps or is there extension stuff that's needed. [11:40:13] Damianz: huh [11:40:27] I just found that problem is not really in backups but in sql server [11:40:31] so I am fixing it now [12:07:28] wow I never saw such a broken server as this one [12:07:37] Damianz: it's completely borked [12:07:54] BORKED [12:08:01] I just purged mysql away, and installed it back, created mysql db now I am can't even create a table [12:08:08] Just rm -rf it, re-install mysql and re-import the data from dumps? [12:08:15] everytime I try to create any table it ends up in datafile corrupted state [12:08:20] o.0 [12:08:21] Damianz: that's what I did [12:08:29] but I can't even do create table [12:08:29] Sure the disk isn't wanked [12:08:33] no idea [12:08:37] I purged it away [12:08:43] now I download latest tarball from mysql [12:08:46] Hmmm [12:08:50] I try to compile my own version [12:08:57] Are you doing this on gluster storage or local disk? [12:09:02] gluster [12:09:12] It's weird because the project storage should never have had an issue to start with... [12:09:14] but I can't create a single table on gluster [12:09:35] maybe the gluster fuck the data? [12:09:49] because create table query instantly fail writing to disk [12:09:49] Maybe but that cluster never had an issue [12:10:00] Silly question - if you remove all the crap out of my.cnf does it work? [12:10:07] which crap [12:10:08] Innodb can have major issues with some my.cnf stuff [12:10:11] when I purged [12:10:15] it removed all stuff in /etc [12:10:20] but on next install it didn't create it [12:10:26] so I had to recover backup of my config [12:10:31] Hmm, then you would have been re-installing to /var/lib/mysql not project data? [12:10:36] yes [12:10:45] but in my backup I already had /data/project [12:10:49] in config [12:10:55] it never created proper config in /etc [12:11:04] probably aptitude remembers it was once installed or dunno [12:11:10] but I did purge mysql-server [12:11:16] Hmm, I'd try with a blank my.cnf and if not just make a new box and start clean as that server is dodgy as hell [12:11:25] how do I create blank my.cnf [12:11:38] I downloaded the tarball for this [12:12:03] I will try to compile it from scratch, who knows what apt does [12:12:23] if it fail again [12:12:25] gluster is fucked [12:14:23] I don't think compiling will fix anything [12:16:26] I think it will [12:16:34] because I will have all install scripts [12:16:45] so I can install it myself [12:17:07] I want to remove all mysql stuff and start from scratch [12:17:13] apt doesn't let me do that [12:18:43] * jeremyb spies a mutante many hours ago [12:19:12] * jeremyb waves from Berlin. odder's here down the hall [12:19:33] hi jeremyb [12:19:40] hi! [12:20:33] * jeremyb is just catching up on ~40 hrs of IRC backlog ;) [12:20:38] !logs [12:20:38] http://bots.wmflabs.org/~wm-bot/searchlog/index.php?action=search&channel=%23wikimedia-labs&query=$1 [12:20:47] !logs jeremy [12:20:48] http://bots.wmflabs.org/~wm-bot/searchlog/index.php?action=search&channel=%23wikimedia-labs&query=jeremy [12:20:54] :P [12:21:02] * jeremyb has /lastlog and awaylog ;) [12:21:12] this is better [12:21:17] you can just filter what you need [12:21:20] XD [12:21:27] !logs money [12:21:27] http://bots.wmflabs.org/~wm-bot/searchlog/index.php?action=search&channel=%23wikimedia-labs&query=money [12:21:37] doesn't return mucc [12:21:40] that's false, /lastlog wins [12:21:44] meh [12:21:49] but thanks anyway! [12:21:53] I use only tail [12:21:55] for last log [12:23:55] ^demon: are you going to replace message [12:24:14] this patch, improvement needs [12:24:24] or whatever yoda would say [12:24:31] <^demon> Yes, but I haven't picked one of the suggestions yet. [12:24:35] <^demon> And I need Ryan [12:24:38] I like Yoda's [12:24:53] much to learn, you still have, padawan [12:26:17] ^demon: it's not possible to have more than one option for the same point value? [12:26:25] or even that questions needs ryan? ;) [12:26:47] <^demon> I'll show you what the table looks like. [12:27:33] <^demon> The answer to your first question is: No, I don't believe so. [12:27:58] I want to diffentiate between "the patch is correct but not something that we want to do" and "the patch has an error or needs improvement". (of course it can be both simultaneously: something we don't want and the wrong way to do it if we did want it) [12:28:06] <^demon> And no, I don't need Ryan for that :) I can actually make the change, but it needs someone to restart gerrit which I lack the ability to do. [12:28:22] oh, plenty of people to do that ;) [12:28:39] <^demon> I'm not going to harass a random root at 8:30am :p [12:28:39] also, I wonder about the gerrit terminology for e.g. SUBMITTED [12:28:52] ^demon: well, in 6 hrs maybe... [12:29:03] <^demon> SUBMITTED is a weird state between OPEN and MERGED. [12:29:14] it seems that SUBMITTED is not pushed for review but rather SUBMITTED is queued for merge [12:29:22] <^demon> In practice, you only see it when there's still an unsatisfied dependency but you submitted. [12:29:28] <^demon> Yeah [12:29:40] so, if we use that definition when parsing "prefer you didn't submit this" it's less of a personal statement? [12:30:04] That's right [12:30:27] <^demon> I don't think it's meant to be personal, and it doesn't bother me really. [12:30:29] SUBMITTED is only used if it's approved to be merged but cannot be merged right now due to dependency problems [12:30:36] <^demon> But it's easy to change so I'm willing to bikeshed a bit. [12:30:37] not that i think it's reasonable for people to be expected to understand SUBMITTED [12:31:09] <^demon> Most people won't see/understand the SUBMITTED state, so "prefer you didn't submit that" appears to refer to them. [12:31:19] <^demon> And less to the person doing SUBMIT -> MERGED [12:31:21] right [12:31:38] so, "i'd prefer you didn't merge this"? [12:31:58] i'd prefer this wasn't merged in it's current form? [12:32:26] <^demon> https://bugzilla.wikimedia.org/show_bug.cgi?id=35406#c1 is what the table currently looks like [12:32:40] <^demon> PK is (category_id,value) [12:33:32] what's category_id? [12:33:38] if i wanted a publically accessible demo wiki on a labs instance, what would i have to do? [12:33:52] can i get a public ip somehow? or is there a proxy infrastructure? [12:34:05] Daniel_WMDE: yes, yes, yes [12:34:14] ^demon: what test instance? on labs? [12:34:19] (btw: i just successfully logged into my instance for the first time, yay!) [12:34:21] <^demon> That's what's on production [12:34:26] Daniel_WMDE: you can even have a subdomain for free ;) [12:34:32] jeremyb: cool :) can you give me a pointer? [12:34:37] ^demon: comment 2? [12:34:38] <^demon> On labs it currently has "THIS IS THE WORST PATCH EVER!!!" for CRVW,-1 [12:34:41] Daniel_WMDE: yeah [12:34:47] haha [12:35:10] Daniel_WMDE: btw, coming for a meal sometime? [12:36:16] * jeremyb digs... [12:36:32] Daniel_WMDE: https://labsconsole.wikimedia.org/wiki/Help:Addresses [12:37:03] Daniel_WMDE: some of those things you can do yourself, some you probably can't (but idk what your bits are) [12:37:22] i'm not a cloudadmin, just a project owner [12:37:25] uh... [12:37:34] meal sounds good... when and where? and what and why? [12:37:52] Daniel_WMDE: no particular reason... just cause we're in town ;) [12:38:24] (alternatively: beer!) [12:38:29] :) [12:38:40] you are in berlin? until when? [12:38:48] leave early tuesday [12:38:48] i'm not here for the chapter's conf myself... [12:39:08] I thought you moved to berlin but now I'm not sure [12:39:13] ok, so... that leaves tonight or monday. i'm meeting with multichil tonight [12:39:25] oh, ha [12:39:27] want to join us? where are you staying? [12:39:38] i hadn't a clue he was in town [12:39:48] so much the better :) [12:39:49] hostel same as everyone else [12:40:33] <^demon> jeremyb: I don't think there's a need to separate "should not do this" and "improve this before we do this." You shouldn't give an unqualified -1 review anyway, so your comments should be telling. [12:40:42] <^demon> Not really different from fixme + a comment on CR. [12:41:11] <^demon> fixme + "Please fix A, B and C with this, but otherwise ok" [12:41:12] hrmmmmm [12:41:20] we could just call -1 fixme? [12:41:23] <^demon> and fixme + "This is the worst idea ever, revert immediately" [12:41:26] * jeremyb looks at gerrit for a min [12:41:52] althought 0 and +1 can also be fixmeish! [12:42:07] jeremyb: i'm not involved with wmconf, so i don't know where the hostel is... is it the baxpax? [12:42:15] Daniel_WMDE: yup [12:42:37] ah, ok... [12:43:22] <^demon> jeremyb: True. Sometimes I'll say "+1, please fix this minor issue and resubmit but otherwise ok" [12:43:43] jeremyb: i can ping you later, once i'm in touch with multichill, so we can decide on where to go [12:44:02] he's staying somewhere else, but i think also close to the U1. [12:49:55] petan: btw, I don't think IPv6 is so simple but maybe andrewbogott knows more [12:50:22] IPv4 addresses are doled out (assigned) by nova [12:51:03] idk exactly how it works but presumably nova has been taught how to dole out v6 addresses. also both need firewall configs managed by nova [12:51:23] and all therefore by OSM. OSM can be fixed, nova is more of a mystery [12:51:49] * Damianz eats jeremyb and notes he tastes like chocolate [12:51:50] [[special:NovaAddress]] is being quite slow for me [12:51:57] Damianz: {{fact}} [12:56:24] ok, [[special:novaaddress]] is fixed i guess? [12:56:29] Daniel_WMDE: how did you fare? [12:57:20] with the addresses? din't follow up on it yet. busy with other stuff. setting up the instance isn't urgent, just something i'm poking at [12:57:25] thanks for your help, though :) [12:58:05] well, it seems to offer reasonable links ("allocate IP" and "add host name") both of which it lets me try to do [12:58:16] jeremyb: I pointed out to that we don't need to use nova at all [12:58:23] .... [12:58:25] why just not to enable ipv6 and ignoring nova [12:58:30] petan: {{fact}} [12:58:37] it doesn't matter it doesn't support it [12:58:41] petan: we should use vmware? [12:58:44] instances run under kvm [12:58:50] petan: you really need to elaborate [12:59:09] nova is using kvm to run instances, and in kvm you can set up ipv6 using nat [12:59:22] only what you need is to enable ipv6 on host system [12:59:35] nova would see instance as using ipv4 only [12:59:40] so every host with IPv6 should be manually tweaked? [12:59:42] even if it was using ipv6 [12:59:46] yes [12:59:55] it should be only for deployment project for now [12:59:59] IMHO it's not worth it [13:00:07] you could maybe use a tunnel broker? [13:00:28] I mean, enabling ipv6 on deployment has higher priority than making nova support it [13:00:40] so we could enable it on deployment for now manually [13:00:43] and later handle the rest [13:01:05] but of course, we can just mess up with nova now, and test ipv6 on production if you prefer that [13:01:11] :D [13:02:01] re seems to offer reasonable links @ 12:58:04 UTC (Daniel_WMDE): but that's only if you can get it to show you anything at all other than the "no credentials" msg or an endless spinning [13:02:13] I didn't watch it [13:02:17] what were you talking about [13:02:30] watch what? [13:02:34] irc [13:02:42] I just opened the window [13:02:43] huh? [13:03:02] < jeremyb> re seems to offer reasonable links @ 12:58:04 UTC (Daniel_WMDE): but that's only if you << this was to me? [13:03:07] 29 12:58:04 < jeremyb> well, it seems to offer reasonable links ("allocate IP" and "add host name") both of which it lets me try to do [13:03:23] ah this can surely wait [13:03:45] I mean, enabling ipv6 on this dialog isn't probably urgent matter [13:03:56] if that's what you meant [13:04:12] those lines have not a thing to do with IPv6 [13:04:17] aha [13:04:38] Special:NovaAddress doesn't load at all for me [13:04:40] or at least Daniel_WMDE hasn't stated a desire to use IPv6 [13:04:47] it just spins [13:04:51] Daniel_WMDE: that's normal [13:04:57] shut down browser [13:05:00] open it again [13:05:06] it has something to do with session mess [13:05:07] ....no. [13:05:14] Daniel_WMDE: see what I just said? ;) [13:05:22] there is a problem with sessions in console [13:05:30] sometimes it happen that your session is stuck [13:05:30] no need to shut down browser [13:05:38] you won't be able to open any page on console [13:05:41] i could kill the cookies [13:05:44] unless you clear you cache / cookies [13:05:46] but i'm not playing with that now [13:05:48] switch browser [13:05:58] periodically you need to log out and back in [13:06:00] ok I just suggested what looked easiest for you to do [13:06:04] no cookie clearing needed [13:06:09] or that [13:06:18] problem is that you might be unable to open logout page [13:06:28] in that case, you need to kill cookies etc [13:06:31] petan: *never* had that [13:06:31] or shutdown browser [13:06:34] I did [13:06:39] "Failed to allocate new public IP address. " [13:06:40] in previous version of console [13:06:53] Daniel_WMDE: you must ask mutante or andrewbogott for it [13:07:02] public ip is restricted [13:07:07] * jeremyb checks on Daniel_WMDE's privs [13:07:10] only labs admins can give you that [13:07:31] brb 20 min [13:09:05] (would be nice if we could see all the privs in one place instead of having to look at multiple places) [13:09:47] Daniel_WMDE: wikidata-dev or ? [13:09:55] yes [13:11:57] petan: Poke me when you are back :) [13:19:17] Damianz: around? [13:19:20] methecooldude: fgagarhg [13:19:55] * Damianz hands petan a knife to poke methecooldude with [13:20:07] I found that when I move db to local storage [13:20:09] it works [13:20:13] o.0 [13:20:13] when it's on gluster it doesn't [13:20:22] Must be something fucky with glsuter then [13:20:25] yes [13:20:27] surely is [13:20:42] I think there is something what is disallowed and mysql needs it [13:20:50] something like locks on files etc [13:21:00] or whatever [13:21:06] it just doesn't work on gluster [13:21:37] so I am moving back to local storage [13:22:02] are you getting an error? [13:22:06] who [13:24:35] petan [13:24:43] ^demon made a choice! ;) [13:25:19] petan: See PM [13:28:25] is andrewbogott SF? [13:28:39] No [13:28:45] He's... somewhere in the US [13:29:16] so, he's probably east of Pacific but not sure where [13:29:28] (/me is trying to judge if awake ;) [13:29:28] <^demon> Well only 50 possible states. Probably can cross off a few like Alaska or North Dakota. [13:29:53] I think he's in Central Time [13:29:58] I have this feeling that he's in the middle somewhere [13:30:57] <^demon> Speaking of timezones, I'd like to bring up the issue of DST again. [13:31:08] hah, what context? [13:31:21] <^demon> How it's a huge pain in the ass. [13:31:26] It is [13:31:31] I switched to DST *twice* this year [13:31:32] paravoid: you around? [13:31:44] i only did once this year. twice last year [13:31:55] Europe switches two weeks after the US, and I my flight was timed just right to catch both switches [13:32:05] * jeremyb too last year [13:32:28] i missed it by ~3 days this year [13:32:30] <^demon> http://en.wikipedia.org/wiki/File:DaylightSaving-World-Subdivisions.png - how come we only do this silliness in North America (mostly), half of Brazil, Europe, and 1/3 of Australia? [13:33:06] Well it doesn't make sense close to the Equator [13:33:06] RoanKattouw: btw, seen my new favorite tool? (new favorite but it's not actually a new thing) http://dpaste.com/718099/plain/ [13:33:22] In places like Singapore, the sun rises at 7am and sets at 7pm year-round, give or take a half four [13:33:24] *hour [13:34:05] <^demon> Russia is decidedly not near the equator, and they gave it up ;-) [13:34:22] And of course Israel switches on a Friday, lol [13:34:52] China also didn't want to have to deal with timezones [13:35:06] <^demon> Which I think is fucking brilliant on the part of the chinese :P [13:35:07] well they do it the night before shabbat so you don't change your clock during shabbos [13:35:21] There "should" be a 1-2 hour difference between the ends of the country, but there isn't [13:35:22] (err, shabbat and shabbos are synonyms) [13:35:25] <^demon> RoanKattouw: You know I hate timezones too :) [13:35:42] hehe [13:35:46] * RoanKattouw doesn't hate timezones so much [13:35:48] They're cool! [13:35:57] * jeremyb doesn't mind timezones [13:36:11] <^demon> UTC for everyone :) [13:36:21] btw, alexanderplatz has a world clock with cities hardcoded. no DST taken into account [13:36:42] also, I have to check how it handles india (UTC+5.5) [13:37:03] <^demon> Half-timezones are kind of silly. [13:37:10] Look up the TZ for Bhutan [13:37:55] <^demon> UTC+6, no DST [13:37:57] hrmm, Jeff_Green seems unidle? [13:37:57] Sorry, I mean Nepal [13:38:32] <^demon> UTC+5:45 [13:38:35] <^demon> What in the hell? [13:38:38] jeremyb: what's up? [13:38:57] <^demon> So you drive over the border from India -> Nepal, and you have to wind your watch forward by 15m? [13:39:01] <^demon> That's just stupid. [13:40:05] Jeff_Green: hey, quickish favor(s): 1) check the wikidata-dev project for the current public IP limit value (but I'm not sure where to look... under manage projects?) and 2) bump it up by one if it's now 0? [13:40:27] Jeff_Green: this is for WMDE (Daniel_WMDE) [13:40:44] would be really nice if the count was exposed somewhere I could check... [13:41:04] jeremyb: thanks :) [13:41:15] yea, having at least one public ip would be very helpful [13:41:19] I'm not aware of a web UI knob for this [13:41:24] more is better :) [13:41:39] Jeff_Green: what do you see for the project under manage? [13:41:49] and the [[Help:addresses]] doc needs to say where to check and how to change it [13:42:13] manage projects appears to be all about members and roles [13:42:48] i suspect ryan is still managing ip quotas etc. via CLI [13:42:55] orly [13:43:15] i assumed I couldn't see it because I wasn't powerful [13:44:44] I'm not *positive* this is the case but last time I asked about instance quotas he disappeared behind the Oz curtain and did magic [13:44:44] still looking though . . . [13:45:09] hah [13:45:40] Waiting for labsconsole.wikimedia.org... [13:45:49] and so we wait . . . and wait . . . and wait . . . [13:46:11] welcome to the club [13:46:17] Jeff_Green: Works fine for me [13:46:54] methecooldude: often depends what page you hit [13:46:59] methecooldude: some of the tools are slower than others, I was waiting for Manage Addresses [13:47:23] Ah, there we go [13:47:30] <^demon> There's a bits outage in Europe, might be causing the labsconsole slowness for some people. [13:47:43] seems unlikely [13:47:54] to me [13:48:00] that's an unfortunate cross-dependency [13:50:31] sorry--i don't see a quick ui tweak for ip allocations. i might be able to figure out how to do it via CLI from my scattered notes in a bit [13:51:46] sure; the docs I see ([[help:addresses]]) are too vague and I somehow never considered it wasn't a part of OSM [13:52:01] thanks for trying anyway. we can try again in a few hrs with andrewbogott [13:52:44] paravoid: nvm [14:00:46] meh [14:01:01] !addresses [14:01:05] what's wrong on that jeremyb [14:01:11] if it's wrong, fix that [14:01:26] !broken is oh no, so fix it! dam you [14:01:26] Key was added! [14:01:45] jeremyb, petan, Daniel [14:01:49] hello [14:01:56] Daniel_WMD: I am awake, but about to go to breakfast. [14:02:03] petan: Is there enough local storage for the db to live there for now? [14:02:05] andrewbogott: I found out the issue [14:02:09] yes it is [14:02:20] + 10gb of free space [14:02:25] problem is with speed [14:02:30] not space [14:02:37] andrewbogott: do you know anything about how to up the floating IP limit for a project? or check the current limit? [14:02:44] Didn't a backup of the db take 90g though? [14:02:50] jeremyb: I do [14:03:01] andrewbogott: no, only 70gb [14:03:01] petan: do tell [14:03:06] jeremyb: We only have ~10 floating IPs for all of labs. So we are very stingy with them. [14:03:06] let me find it [14:03:17] andrewbogott: forever? [14:03:23] andrewbogott: this is for WMDE... [14:03:31] (wikidata) [14:04:48] There are simultaneous plans to a) set up a reverse-proxy so that more instances can have a public http address b) find more ips somehow c) switch to ipv6 someday [14:04:58] but none of those will happen in the next few weeks :( [14:05:43] I'm reluctant to hand out a public IP w/out Ryan's say-so. do you mind emailing him and cc'ing me? I think he's still reading his mail despite being on holiday. [14:05:47] ok, well do you at least know how to? [14:06:35] and what if Daniel_WMDE agrees that it might be only temporary and he'll know in ~1 week whether or not he can rely on not getting it taken away? [14:07:01] (ryan's back in a week?) [14:07:16] Yeah, in a week. [14:07:24] jeremyb: https://labsconsole.wikimedia.org/wiki/Help:Nova-manage [14:07:26] here it is [14:07:44] petan: wooot [14:07:52] nova-manage project quota --project=blah [14:07:57] will show you the limit for blah [14:08:15] andrewbogott, jeremy: wikidata needs at least one publically accessible test wiki in about a week. [14:08:25] we can wait until then, but not much longer [14:08:38] Daniel_WMDE: any chance that development could merge to existing one? [14:08:41] i don't need an ip, a proxy based solution is fine with me [14:08:41] like deployment etc [14:08:50] petan: no. [14:08:53] we have a working infrastructure there [14:08:56] Daniel_WMDE: would you run your own load balancer to make multiple domains out of a single IP? [14:08:57] ok -- hopefully we'll hear back from Ryan in a day or two. [14:09:02] setting up a new wiki for you is simple [14:09:32] You won't, he's on vacation [14:09:35] petan: we'll also do things with deployment, but we definitly needs something for testing in earlier stages of development [14:09:47] <^demon> He'll be back Monday, right? [14:09:48] Daniel_WMDE: but that you can have there as well [14:09:50] we have het there [14:09:51] Yes [14:09:59] so that you can create a wiki using different branch [14:10:08] then i don't understand what deployment is really for :) [14:10:27] it's primarilly for testing the sw before deployment but it has much bigger potential [14:10:37] i thought that was for testing in the full deployment environment? for stuff in the deployment pipeline? [14:10:49] yeah, i think deployment's not a match here [14:10:50] it's configured just as production is, so... we can do magic there [14:10:56] petan: are you in the progress of rebuilding the db still? [14:11:02] andrewbogott: yes [14:11:04] it's running [14:11:10] petan: i'm fine with whatever gives us full control over the wiki setup, extensions, ssh access, etc [14:11:20] petan: but ssh access is managed at the project level, no? [14:11:28] wikidata is for stuff that won't be deployed for at least 5-10 weeks from now [14:11:31] petan: would you expect me to get page loads yet? [14:11:34] and you don't really want 10 new devs in thedeployment project?... [14:11:37] (maybe double that eve) [14:11:38] Daniel_WMDE: let'w wait for someone to tell you if you can have ip or not, then we can check out this option [14:11:39] even* [14:12:02] Daniel_WMDE: as long as they don't break other stuff, I don't really care :) [14:12:03] i could ask erik :) [14:12:10] hah [14:12:20] petan: they are bound to. [14:12:30] I could ask a few board members ;-P [14:12:32] * Daniel_WMDE wonders if erik is in berlin for the chapter's conf [14:12:38] I just thought that having one production like cluster for testing is better than having 20 production like clusters [14:12:39] * jeremyb goes for coffee break [14:12:43] much cheaper at least [14:12:51] petan: i don't need a production like cluster [14:12:53] not yet [14:12:55] we need that too [14:12:57] deployment project has more than 12 servers atm [14:13:01] but first we just need a vm for testing [14:13:24] you can have a vm for testing, problem is with ip [14:13:36] at some point we could merge huggle wa with huggle [14:13:40] that would give us 1 free ip [14:13:44] Daniel_WMDE: looks like no erik [14:14:15] andrewbogott: if I release IP, can you give it to Daniel_WMDE [14:14:15] petan: so, set up a nginx to proxy :) [14:14:32] because I have one I don't need atm [14:14:37] bbiab [14:15:08] petan: maybe, but unless this is a huge emergency I would rather this all go through ryan. [14:15:22] * andrewbogott -> breakfast [14:15:24] this is important, but not urgent [14:15:30] we'll resolve it next week. [14:15:34] no need to rush it [14:15:34] right [14:29:58] semi back [14:43:48] !log deployment-prep petrb: recovery is finished, not data lost [14:44:02] !log deployment-prep petrb: 1.sql apache apacje backup CentralAuth check_puppet fdgaggrehresjarejas fix global index.html InitialiseSettings.php IS.php list log motd mysql MySQL-client-5.5.22-1.linux2.6.i386.rpm MySQL-devel-5.5.22-1.linux2.6.i386.rpm MySQL-embedded-5.5.22-1.linux2.6.i386.rpm MySQL-server-5.5.22-1.linux2.6.i386.rpm MySQL-shared-5.5.22-1.linux2.6.i386.rpm MySQL-test-5.5.22-1.linux2.6.i386.rpm projects rotd site site_b site [14:44:17] lmao [14:44:30] * :D [14:45:00] ok Damianz you were right about that command can leak stuff you don't want to leak, at some point [14:45:29] I just publicly leaked my $HOME [14:47:19] is it some kind of sudo command which posts to the bot? [14:49:47] no sudo [14:49:50] just log [14:49:55] log * typo [14:50:04] however * in shell expand to list of files [14:50:09] that's why [15:23:20] labs-nagios-wm: what's up [15:23:24] it doesn't report [15:24:07] this is a flood test [15:24:12] hm [15:24:16] this is a flood test [15:24:26] yeh it looks like it should work [15:24:58] petan: no data lost* [15:25:00] ? [15:25:15] yes [15:25:22] that's what I posted as second log [15:25:25] with * [15:25:28] :P [15:25:35] heh [15:25:42] * in shell does a lot [15:26:09] \* or '*' [15:26:12] anyway I need to go [15:26:13] I know [15:26:17] later [15:26:18] PROBLEM Free ram is now: WARNING on deployment-web2 deployment-web2 output: Warning: 15% free memory [15:26:20] see you [15:26:22] ouch [15:26:28] deployment site is getting down [15:26:35] 1 server is down [15:26:39] others are coming [15:26:43] someone is ddosing us [15:31:18] PROBLEM Free ram is now: CRITICAL on deployment-web2 deployment-web2 output: Critical: 5% free memory [15:36:18] RECOVERY Free ram is now: OK on deployment-web4 deployment-web4 output: OK: 33% free memory [15:36:18] RECOVERY Free ram is now: OK on deployment-web2 deployment-web2 output: OK: 33% free memory [15:39:28] PROBLEM host: deployment-web5 is DOWN address: deployment-web5 CRITICAL - Host Unreachable (deployment-web5) [15:41:28] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: Critical: 5% free memory [15:46:28] PROBLEM Free ram is now: WARNING on deployment-web deployment-web output: Warning: 6% free memory [15:56:13] petan: site looks good! [15:58:35] petan: actually, I spoke too soon... looks like a few files still never load. Any idea what's up with that? (view the commons front page to see.) [15:58:57] oh, I guess that could relate to all those nagios warnings :( [15:59:13] methecooldude: hey [16:04:05] Ready: Just before petan left he said 'someone is ddosing us' regarding deployment. Can you help me understand if that's happening and/or how to fix it? [16:04:11] s/Ready/Reedy [16:05:25] Depending on the actual nature of it, we'd probably be best pinging Leslie when she's in the office [16:05:36] So she can block the user(s) at the core routers [16:05:47] ok [16:05:53] I'll wait for her then, thanks. [16:06:50] btw -- I'm missing a lot of background information. Who actually set up the deployment cluster, installed mediawiki, etc? [16:07:20] Everyone seems to treat it as though it were created as an act of god. [16:07:26] Haha [16:08:26] Quite a few people have had a hand in it from time to time, AFAIK petan set up the basis of it, I know hexmode has had some involvement, and as have a few other volunteers [16:10:28] PROBLEM host: deployment-web5 is DOWN address: deployment-web5 CRITICAL - Host Unreachable (deployment-web5) [16:11:08] Reedy: fair enough. thanks. [16:16:31] alejrb: Hey, sorry, didn't see yuou there :P [16:39:17] andrewbogott: I've pinged leslie [16:40:09] <^demon> It's more fun to imagine things just appearing out of nowhere. [16:41:28] PROBLEM host: deployment-web5 is DOWN address: deployment-web5 CRITICAL - Host Unreachable (deployment-web5) [16:44:23] hey petan [16:44:28] so you think there's some issues ? [16:47:39] LeslieCarr: [17:04:07] Ready: Just before petan left he said 'someone is ddosing us' regarding deployment. Can you help me understand if that's happening and/or how to fix it? [16:48:13] those things are so hard to tell --- i'm gonna check the tcp counters on virt0-3 [16:48:20] okay, http://bots.wmflabs.org/~petrb/logs/%23wikimedia-labs/20120329.txt [16:48:22] see if there's a lot of unacked packets or anything [16:48:26] thnanks [16:48:28] time stamp was [15:26:43] [16:48:31] np [16:48:37] ah [16:48:40] interesting [16:49:08] well that's a good start, all on the deployment servers [16:50:16] 03/29/2012 - 16:50:16 - Creating a home directory for lcarr at /export/home/deployment-prep/lcarr [16:51:15] 03/29/2012 - 16:51:15 - Updating keys for lcarr [16:53:53] LeslieCarr: deployment-web5 seems to've died altogether (which it seems to do about once per day.) I'll reboot it unless that will mess up whatever you're looking at now. [16:54:01] rebooting is ok [16:54:36] Can you tell if deployment is getting swamped with traffic or if it's just lame? [16:56:28] RECOVERY Free ram is now: OK on deployment-web5 deployment-web5 output: OK: 76% free memory [16:56:38] RECOVERY host: deployment-web5 is UP address: deployment-web5 PING OK - Packet loss = 0%, RTA = 0.81 ms [16:56:50] it doesn't look swamped - there was a few out of memory errors [16:56:54] these are pretty small instances [16:57:06] i sstuff like this normal ? wikiquote:80 10.4.0.17 - - [29/Mar/2012:16:56:00 +0000] "GET /wiki/Posebno:GlobalniKorisnici HTTP/1.0" 200 6558 "" "EC2LinkFinder" [16:58:28] RECOVERY SSH is now: OK on deployment-web5 deployment-web5 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [16:58:28] RECOVERY Disk Space is now: OK on deployment-web5 deployment-web5 output: DISK OK [16:58:28] RECOVERY Current Load is now: OK on deployment-web5 deployment-web5 output: OK - load average: 0.58, 0.37, 0.15 [16:58:28] RECOVERY Current Users is now: OK on deployment-web5 deployment-web5 output: USERS OK - 0 users currently logged in [16:58:28] RECOVERY Total Processes is now: OK on deployment-web5 deployment-web5 output: PROCS OK: 122 processes [16:58:33] RECOVERY dpkg-check is now: OK on deployment-web5 deployment-web5 output: All packages OK [16:58:37] Dunno, but 'EC2LinkFinder' sounds like an openstack component. [16:59:49] LeslieCarr: Do you have the superpower of being able to figure out why web5 died? [17:00:09] not really, i'd do the same thing as you, looking through var/log/messages or syslog or checking out apache logs [17:00:31] 'k [17:01:28] Deployment is clearly underpowered and unreliable, but I'm not clear on how much time should be spent on sorting it out. (And I am pretty clear that I'm not the right person to do so.) [17:02:28] it doesn't feel like a network ddos at all (not lots of syn packets flooding in) - that doesn't mean that it's not being overused for the amount of resources it has though [17:05:09] Certain links continue to fail in the way they always have. I wonder if the overload is self-inflicted somehow. [17:06:40] anyway, thanks for looking! [17:29:18] PROBLEM Puppet freshness is now: CRITICAL on aggregator1 aggregator1 output: Puppet has not run in the last 10 hours [17:29:18] PROBLEM Puppet freshness is now: CRITICAL on analytics analytics output: Puppet has not run in the last 10 hours [17:29:18] PROBLEM Puppet freshness is now: CRITICAL on asher1 asher1 output: Puppet has not run in the last 10 hours [17:29:18] PROBLEM Puppet freshness is now: CRITICAL on backport backport output: Puppet has not run in the last 10 hours [17:29:18] PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 bastion-restricted1 output: Puppet has not run in the last 10 hours [19:09:44] petan: ping me when you have a minute? [21:26:13] 03/29/2012 - 21:26:13 - Creating a home directory for owen at /export/home/bastion/owen [21:27:13] 03/29/2012 - 21:27:12 - Updating keys for owen [21:34:13] 03/29/2012 - 21:34:13 - Updating keys for owen