[01:10:04] * legoktm pokes Damianz [01:10:07] cluebot is down? [01:28:19] phe: You can still use qsub with happy fun -l h_vmem=$foo [01:28:39] phe: jsub is there for the newbies and the peeps who don't want to bother with qsub arcana. [08:24:11] legoktm is it? [08:24:53] @notify addwork [08:24:53] You already requested this user to be watched [08:25:18] It is [08:25:23] let me check [08:25:24] Since a loooong time ago [08:25:32] I have no idea how to restart it because there is no guide [08:25:36] !botsdocs [08:25:36] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [08:26:03] last edit was [08:26:04] 20:25, 18 March 2013 (diff | hist) . . (-25)‎ . . m Nino Bravo ‎ (Reverting possible vandalism by 212.231.247.181 to version by Addbot. False positive? Report it. Thanks, ClueBot NG. (1561812) (Bot)) (current) [rollback: 1 edit] [rollback] [vandalism] [08:26:39] !botsdocs is https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [08:26:39] This key already exist - remove it, if you want to change it [08:26:46] erm [08:26:51] !botsdocs del [08:26:51] Successfully removed botsdocs [08:26:54] !botsdocs is https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [08:26:54] Key was added [08:47:32] * Beetstra does not understand anything of perl's setrlimit ... :-( .. his modules don't crash when he tests them with low values [08:52:08] * Beetstra looks at Coren|Sleep [08:52:33] Coren|Sleep .. I think that the RSS memory limits are not implemented and hence ignored by the OS [09:23:15] !log deployment-prep created sudo policy for jenkins-deploy user. That is the user for the Jenkins slave running deployment-bastion [09:23:18] Logged the message, Master [09:37:41] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: Immediate - normal) [Bug 45084] autoupdate the databases! - https://bugzilla.wikimedia.org/show_bug.cgi?id=45084 [10:01:42] !log bots root: mysql: btrfs filesystem resize -10G /db [10:01:44] Logged the message, Master [10:03:07] !log bots root: running online fsck [10:03:08] Logged the message, Master [10:05:28] o.O? [10:13:04] !log bots root: created /logs_db for binlogs [10:13:06] Logged the message, Master [10:13:33] !log bots root: houston filesystem /db is broken - new kernel is needed to fix it [10:13:34] Logged the message, Master [10:13:55] !log bots root: mysql is running fine though - just btrfsck is crashing [10:13:57] Logged the message, Master [10:20:17] !log bots root: restart of mysql is needed to update configuration [10:20:19] Logged the message, Master [10:28:51] !log bots petrb: updated some packages on sql [10:28:53] Logged the message, Master [10:31:00] btrfsck /db [10:31:03] awesome XD [10:31:20] Damianz that filesystem is kind of borked :P [10:31:29] I mean so borked that fsck is crashing [10:31:30] :( [10:31:37] but on other way it seems to work [10:31:44] mysql is not reporting any issues [10:31:52] so maybe this is problem with some metadata [10:31:56] which aren't important [10:32:13] btw I was amazed by speed of btrfs online shrink [10:32:17] took like 2 seconds [10:32:18] :P [10:32:49] we really need 3.8 kernel on that box [10:35:07] lol btrfsck is crashing is similar way even on my laptop with 3.8.1 kernel [10:35:15] I think this is some common problem then [10:46:15] petan, it seems that the box does not care about the memory-limits of the processes [10:46:18] Damianz if you want to move binlogs we need to schedule outage of sql [10:46:32] Doesn't need to move right now - just stop the disk filling [10:46:32] Beetstra which box? [10:46:38] Damianz sure I already did [10:46:40] whichever [10:46:51] Beetstra did you supply the limit using qsub? [10:46:53] the bot that I am using to test runs on -bnr3 [10:46:58] Beetstra I am really no expert of grid, Coren|Sleep is [10:47:27] Beetstra but there are no real memory limits configured per queue now, so if you need some you would need to specify them per task [10:47:48] I have specified them in perl for the specific modules .. [10:47:48] btw on -bnr3 there is sooooo much free ram atm [10:47:54] not for the whole task [10:47:54] ah [10:48:01] in that case it's problem of perl [10:48:06] maybe there is some library missing? [10:48:18] I just say, 'this perl module is not allowed to use more than 20M' .. but it happily uses 34M [10:48:33] #perl-people say that it is that the box does not care .. [10:48:59] that sounds weird - the box doesn't care because it's not supposed to, there are no memory limits atm [10:49:00] It just does not send a SIGKILL to the processes that violate [10:49:18] it should be that perl module watching its own usage [10:49:25] The script sets them correctly [10:49:36] box will not send sigkill because the box itself has no problem with processes eating lot of ram [10:49:36] No, because it tells the BSD [10:49:57] hmm [10:50:05] maybe it's some system option that needs to be set? [10:50:11] is there some manual for that perl function [10:50:13] I am using BSD::Resource::setrlimit [10:50:25] cpan, BSD::Resource [10:50:34] http://search.cpan.org/~jhi/BSD-Resource-1.2904/Resource.pm [10:50:34] are you sure it doesn't require something that isn't installed? [10:50:51] I installed the perl module in cpan, otherwise it will not run [10:51:02] hmm [10:51:09] maybe we could install it using apt everywhere [10:51:11] what is name [10:51:16] lets wait for Coren [10:51:47] * Beetstra pulls the bucket of ice cold water a bit closer to Coren [10:52:56] it says in the cpan-resource: "Soft or hard limit RLIM_INFINITY means as much as possible, the real hard limits are normally buried inside the kernel and are very system-dependent." [10:52:57] I don't know if coren is perl guy [10:53:16] well, should be the same on python, way of doing this is exactly the same [10:53:17] this is definitely something related to perl and not grid [10:53:32] At least, the code I saw yesterday is quite the same [12:45:31] The sleeper has awakened. Somewhat. (pre-coffee) [12:46:46] !Beetstra [12:46:49] Hey, Coren [12:46:52] Good morning [12:46:59] Woes of memory? [12:47:06] kind of [12:48:10] I have in each module the line "my $success = setrlimit(RLIMIT_RSS, $settings{'xxxx.memlow'},$settings{'xxxx.memhigh'});" [12:48:24] and reading out those values shows that they are set appropriately [12:48:35] But to whatever value I set them, the module never crashes [12:48:43] (perl-code) [12:49:23] in #perl they suggested that the box may not support it? [12:49:39] question is "what" is it not supporting [12:49:45] in terms of UNIX not terms of perl [12:50:07] in a way a good question [12:50:30] RLIMIT_RSS is an odd duck that doesn't behave like most people want. [12:50:35] You want RLIMIT_AS [12:50:36] oh [12:51:06] RLIMIT_RSS is only about locking memory down with madvise() (for things like crypto) [12:51:06] "(virtual) address space" ?? [12:51:13] Beetstra: Right. [12:52:07] But that doesn't get your process killed regardless; hiting a memory limit just makes sbrk() fail (perl, though, dies if it fails to allocate memory IIRC) [12:56:48] the module crashes with "Out of memory!" [12:57:10] !log bots madman: deleting bots-2 [12:57:11] Logged the message, Master [12:57:24] so that works .. in perl [12:57:25] Not very polite error handling, but it's working. [12:57:29] !log bots petrb: previous message was supposed to be logged as me [12:57:30] Logged the message, Master [13:03:04] hmm, interesting .. "libgcc_s.so.1 must be installed for pthread_cancel to work" [13:25:57] Beetstra: That's a symptom of low memory. [13:26:15] Beetstra: Ze limit. She is tooo smalle! [13:26:16] yeah, I knew I was there between the limits [13:26:34] Hey .. Italian accents are for my wife to produce ;-) [13:26:50] *lots* of code is written presuming that memory allocation cannot fail. Lots of lazy programmers. [13:27:35] I see the use of it, so I am trying to implement it now [13:28:05] Oh, I don't mean you. Even in system libraries of language interpreters you see that. [13:28:07] and for the others on the same instance: "Thou shall not pass!!!!" (.. this memory limit) ;-) [13:29:16] It is also for me, learn new things with this [13:31:27] I'm a (realy) old school coder; I still believe in being careful with not wasting core memory and carefully desinging code to keep a low footprint. The kids, these days, they spout the current common wisdom at me "ram is cheap, coder time is expensive", and then waste days tracking down problems. :-) [13:32:48] I am just a hobby-programmer, but writing the linkwatchers is something that needs to run optimally if you want the data in real time [13:32:59] If only the mediawiki software would record the data itself [13:33:23] Beetstra: Why not write an extension for MW? [13:34:04] Jan_Luca .. that would take so much time .. which I don't have [13:34:20] This is something that runs and I just need every now and then an hour or so to tweak [13:36:14] Coren, so now what happens. Say, I have a process that takes currently 50 Mb, with a hard-limit of 250 Mb. Someone else runs a rogue bot .. will that bot run out of memory when it hits the outside of my 250 Mb hard limit, because my process has the right to use that 250 even if it does not? [13:37:49] Beetstra: Nope. On a grid that enforces those limits, the others won't encroach on you because they hit /their/ limit before they can. [13:38:13] Beetstra: the rlimit doesn't allocate, it constrains. [13:38:26] So it takes only one rogue bot to still take down mine [13:39:07] Beetstra: Yes, unless the scheduler applies a limit to everything and doesn't overallocate. [13:39:28] OK [13:39:37] Beetstra: Which is why tools- is configured that way, unsurprisingly enough. :-) [13:41:16] overcommit_memory is set to 2 in tools- instance ? [13:44:25] phe: Yes, that too, but I meant the gridengine sets and enforces vmem-based allocation. [13:47:36] !tooldocs [13:47:36] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help [13:47:49] Coren can you tell me how you configured the limits on grid [13:48:03] or is it documented somewhere [13:48:22] petan: I have, the other day, you just didn't notice. Make h_vmem consumable, and set a limit on the nodes. [13:48:38] nodes? I thought limit is on queue [13:48:53] that would mean that different nodes could have different limits? o.o [13:48:56] petan: You can have limits on either, but h_vmem makes no sense on queues. :-) [13:49:09] ah [13:49:32] ok, so... you specified some default limit? and users can override it? [13:49:36] petan: Yes. You'd obviously put the "real" amount of memory on the nodes (usually, you'd want physical-1G) [13:49:38] by specifying different value? [13:49:53] petan: Right. The default is just convenience. [13:50:08] ok, users can override it how they want or is there a limit too? [13:50:22] so you just set h_vmem to 256mb on nodes? [13:50:42] No, the setting on the node is the total quantity available. [13:50:47] Like, "7g" [13:50:57] ah [13:51:04] so where you set the 256mb? [13:51:53] In /var/lib/gridengine/default/common/sge_request. Default values to there. [13:51:58] ok [13:52:01] s/to/go/ [13:52:35] (simple text file with one option per line so) [13:52:47] -l h_vmem=128m [13:52:49] (on one line) [13:52:53] ok [13:52:57] Err, 256m [13:53:07] 128 was my first attempt. Too small for python. [13:53:32] this should be in puppet variable so that we can use it on staging with same values [13:54:05] petan: Hm. I was planning on just putting the sge_request file in puppet. [13:54:19] we could either put whole config to puppet but that would complicate it to play with it on staging in order to test different conf [13:54:36] because puppet would just override it [13:54:39] whole [13:55:09] petan: Not unreasonable. [14:01:08] btw Coren qtop is working on your grid as well... [14:01:24] or it appears to me to work :P [14:02:26] just cp -r /home/petrb/bin ~/bin [14:05:06] petan: I'll stuff it in /usr/local/bin. Want to go doc it on the help page, then? [14:05:48] sure [14:06:11] I see it needs qdisplayinfo.sh as well. [14:06:30] Do you intend that one to be directly usable as well? [14:06:33] well, we could probably merge it [14:06:45] qdisplay print the info once [14:06:52] qtop refresh it every second [14:07:01] qdisplay has that advantage it works in small screen [14:07:09] while qtop will not display all text in small terminal [14:07:20] qtop is just a wrapper for qdisplay [14:07:30] Hm. Okay, I'll rename it to qdisplay then [14:07:34] ok [14:07:48] Coren: BTW, it would be nice if *all* scripts & Co. have at least a one-liner licence so that there's no ambiguity. [14:08:00] scfc_de: Oh, good point! [14:08:01] feel free to use any licence you want... [14:08:10] I am fine with WTFPL :P [14:08:18] or GPL [14:08:41] They're minuscule. I was thinking CC-0 [14:09:08] I think WTFPL is even more liberal than CC-0 :P [14:09:09] Or DWTFWIWI :-) [14:09:17] that's probably same [14:09:21] just longer name [14:09:40] Yeah, CC-0 is... nicer. :-) [14:09:50] Most of the scripts probably aren't copyrightable at all :-), but it's much nicer to sort this stuff out now then in five years when noone remembers who did what. [14:10:59] Coren what is that "Why" header for [14:11:08] * "How" [14:11:13] 2.1 How [14:11:14] 2.2 Simple utilities [14:11:19] section 2.1 has no text [14:11:48] Yeah, I'm not done yet. I want to give a summary of how submitting jobs work and what happens to them. [14:12:04] ok maybe that qtop could go there in future [14:12:08] no idea where to put docs for that [14:12:18] it should be in "grid status" section or whatever [14:12:23] or grid status overview [14:13:06] * Coren asks WMF legal for the right boilerplate. [14:13:47] petan: bots-bsql01 down? [14:13:55] Damianz since when? [14:14:11] It pings but I can't connect to mysql or ssh to it [14:14:18] o.O [14:14:22] Bot last reverted something 2 hours ago [14:14:38] let me check [14:14:48] I can't ssh either :/ [14:14:48] [611760.713140] INFO: task mysqld:16620 blocked for more than 120 seconds. [14:14:51] [611760.713755] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [14:14:54] *snort* [14:15:07] I think it's a good time to schedule our restart of mysqld then :> [14:15:28] the box probably [14:15:48] !log bots -sql has some troubles - Damianz is responsible for sure :P [14:15:50] Logged the message, Master [14:16:03] wait [14:16:07] don't reboot it yet [14:16:13] I would like to find out what happened [14:16:48] * Damianz gives you 30min [14:16:57] I am there :> [14:16:58] sshed [14:17:10] but still launching local shell [14:17:50] hehe load 143 [14:18:25] That would be why the process is hung [14:18:38] !log bots mysql db ate 31gb of ram [14:18:39] Logged the message, Master [14:18:46] that needs some investigation [14:18:55] I am stopping mysqld now [14:19:08] !log turning off mysqld to fix ram problems and moving binary logs [14:19:09] turning is not a valid project. [14:19:13] !log bots turning off mysqld to fix ram problems and moving binary logs [14:19:14] Logged the message, Master [14:19:56] Anyone currently using the tools- queue? I should reboot to reapply some security updates. [14:20:26] Coren not me and qtop is telling me 0 jobs are running [14:20:38] not that qtop is trustworth :P [14:20:59] Damianz wow it takes some time to kill a process that eats 30+gb ram [14:21:27] !log tools reboot cycle (all instances) to apply security updates [14:21:28] Logged the message, Master [14:21:29] it just doesn't want to die [14:22:20] petan: Ah, the joys of swap. :-P [14:23:04] !log bots killing mysqld with -9 :| [14:23:05] Logged the message, Master [14:23:28] !log bots not that it would really worked :P [14:23:29] Logged the message, Master [14:25:18] Unable to parse the feed from https://bugzilla.wikimedia.org/buglist.cgi?chfieldfrom=-4h&chfieldto=Now&list_id=151044&product=Wikimedia%20Labs&query_format=advanced&title=Bug%20List&ctype=atom this url is probably not a valid rss, the feed will be disabled, until you re-enable it by typing @rss+ bugzilla [14:25:36] These are just "user databases", aren't they? [14:26:28] !log bots root: restarting mysql server with new configuration [14:26:29] Logged the message, Master [14:28:01] !log bots started recovery of innodb.. will take some time [14:28:03] Logged the message, Master [14:28:28] that maria doesn't want to start, seems like data are corrupt [14:28:35] fortunately we have binary logs :> [14:29:43] kill -9 can cause some damage. [14:29:51] Bots back up anyway [14:36:08] !log bots recovery finished, db is back :)) [14:36:10] Logged the message, Master [14:36:55] andrewbogott: hey :-] Your puppet documentation root page seems to use Twitter bootstrap now https://doc.wikimedia.org/ =) [14:37:25] andrewbogott: well at least the main page. The index file is somewhere under integration/docroot.git . I guess we will happily co maintain it [14:37:32] scfc_de be sure I don't use kill -9 unless I have to :/ [14:37:49] that box would die on OOM if I didn't [14:39:04] hashar: I don't know enough about css to know what 'Twitter bootstrap' means... [14:39:13] That front page should probably become a wiki page, someday [14:39:50] andrewbogott: maybe yeah :-] [14:43:40] Damianz I am thinking of killing some swap on bsql01 and using that for lvol2 but not sure if that is a good idea [14:43:58] meh... why we just can't have another virtual disk plugged to instance [15:03:46] !log nagios petrb: restarted ircecho [15:03:47] Logged the message, Master [15:04:22] !ping [15:04:23] pong [15:08:41] Coren: I have added to my list some questions for a sysadmin doc of tools: https://wikitech.wikimedia.org/wiki/User:Jan/Tools#Sysadmin_doc [15:10:13] Jan_Luca: Indeed, those are important questions. [15:11:49] Coren: Maybe petan could add questions he has [15:12:25] Jan_Luca: That would be wise indeed. :-) *hint, hint* [15:36:47] Coren: You write in your documantion only about CGI. Is FastCGI installed, too? [15:38:31] Jan_Luca: suphp; I know of no reliable way to deploy FastCGI in a compartimentalized fashion. [15:40:39] Coren: suexec? [15:42:17] Jan_Luca: That's explorable as a technique. There's also the problem that I would very much prefer to not have long-running processes run on the webservers. It's in the "would be nice - will look into it" slush pile. [15:43:25] * Coren would idealy prefer a scheme by which the fastcgi app runs on the grid, but that would require some unix socket /wedge/ on the webserver. Not impossible, just not trivial. [15:43:59] Coren: But if there are tools using cgi with high access, every time a new cgi process will be created [15:44:16] Hence "would be nice" :-) [15:45:25] Remote FastCGI would work, but the lack of certificates means that insulation is hard. [15:48:11] Coren: Maybe you could ask the Toolserver roots how they have set up FastCGI [15:48:30] Jan_Luca: I already know, by allowing the actual app to run on the webserver. [15:48:46] Jan_Luca: Which I do not want to do here for stability and security. [15:49:09] Coren: But using cgi makes no difference [15:49:33] Jan_Luca: In theory, it's not /especially/ hard to do it; but it takes some effort and it's not clear to me how many people need/use it. [15:50:01] Jan_Luca: Yes, but CGIs are short lived, and do not survive the connection; the issues with persistent daemons are not the same. [15:50:43] Jan_Luca: If you think there is a serious call for FastCGIs, I can arrange it. I just didn't think there'd be that much demand for it. [15:50:44] Coren: When I unterstand you right you want to create a FastCGI daemon on another host like php-fpm? [15:51:06] Jan_Luca: Yes, using the gridengine for creation and dispatch. [15:51:34] Coren: Ah, you want to use proxy_fastcgi, now I get it [15:53:51] Not quite, actually, but the same principle. (proxy_fastcgi is really hard to do right without sysadmin intervention to the apache configs, I want something the maintainers can do themselves) [15:54:29] Coren: A small wrapper that calls the right process? [15:55:09] Right, so that the fastcgi server is started on the grid instead of locally. [15:56:12] Coren: Ok, you're right, that would be a better way to enable FastCGI than my idea [16:05:44] Jan_Luca: Which Toolserver apps use FastCGI? [16:06:20] scfc_de: I don't know but maybe there are some [16:07:53] Jan_Luca: Then I'd suggest that we wait until someone actually requests it :-). [16:08:25] scfc_de: I only want to have a high compatibility to Toolserver :-) [16:12:23] Jan_Luca: I don't. I hope Solaris dies. I hope JIRA dies. I hope all the other cruft that has accumulated over the years dies. I hope for reliable, responsive, usable replicated databases. [16:15:21] scfc_de: I don't mean the thing you listed, I think of webserver envirment, SGE, ... [16:15:40] You're right that some thing should not be migrated [16:20:18] Jan_Luca: But FastCGI that - for the sake of this argument - isn't used, should be migrated? Every astray from a "standard" setup means not only admin work setting it up, but also maintaining it and debugging it. We've seen at Toolserver how a plethora of seldomly used services makes admin work unnecessarily complex. [16:21:42] scfc_de: I thought FastCGI is used because not everybody likes PHP and FastCGI is IHMO better than CGI [16:22:28] Jan_Luca: Do you use FastCGI? :-) [16:23:15] scfc_de: No, because I like PHP. ;-) [16:23:37] I only want to create a nice envirment for everybody [16:23:56] I tried tu use fcgi with pyton on the TS and fallback to cgi as I was unable to get something working [16:24:52] Jan_Luca: Then you probably have some expertise in deploying PHP apps and can help Coren in this regard. [16:25:08] phe: Eh, did FastCGI work or not? [16:25:37] scfc_de, I don't remember if the problem was on my side or on TS side :) [16:27:18] I suppose it's really mostly a matter of "does anyone /use/ it?" [16:27:33] phe: I never touched FastCGI as IIRC a FastCGI process is destroyed after a few seconds so it would only be useful for *really* heavy use while requiring a lot of rewriting the app. Also, it would have required me to set up FastCGI on my local box as well. [16:27:50] scfc_de: I already asked Coren where I can help [16:28:10] I mean yes, I know ostensibly FastCGI is a better design, but unless people actually do use it, the effort of another component isn't a good idea. [16:28:25] The fewer things that are in place, the fewer things that can break. :-) [16:28:54] He answered that at moment the best help would be things like questions for his documentation [16:29:26] Jan_Luca: Sure, but if you find some needed component you have expertise with, help is always welcome. [16:30:35] Jan_Luca: I meant finding any hiccups in the PHP setup that make app deployment more difficult/impossible. [16:31:48] Jan_Luca: (And if you don't find any, that's not a failure on your part.) [16:32:48] scfc_de: At moment I have no good tools to test the system [16:33:24] I help at moment developping MW itself when I have time [16:33:34] Jan_Luca: Don't you maintain one of Magnus' tools, Commonshelper or something like that? [16:34:35] scfc_de: Yes, but since the Commons' admins disabled my bot the usbility is not as I want to [16:36:31] and it needs some review at moment [16:36:32] scfc_de: And CommonsHelper2 would need a replacement for TUSC to enable auto-uploads [16:38:28] scfc_de: And a replacement for TS-I18N ;-) [16:38:57] I have to rewrite some parts to get it running on Tools... [16:39:37] Jan_Luca: I think you should be able to port TUSC 1:1 to Tools (perhaps renaming it) if the wait for OpenID is too long. But I would fix the Commons issues before. A bot that is disabled on its wiki needs some work. [16:41:07] scfc_de: The tools itself works without the bot but it would be nicer when the auto-upload works [16:41:54] And TUSC is a tool of Magnus, he have to copy the database, too [16:43:05] Jan_Luca: Well, as a user, I wouldn't care if /some/ part of a tool works, if overall it fails :-). For Tools User Screening Tool, you would obviously need to set up a new database and new credentials. [17:17:08] gerrit-wm seems to have a problem... [17:27:43] <^demon> Jan_Luca: Known, have a patch up for it. [17:27:49] <^demon> It's restarting each time puppet runs. [17:36:46] Coren, Ryan_Lane, I'm making progress on the gui for service groups. Do you have time to work on the other bits today? [17:37:13] I'm imagining that Ryan_Lane will make the Schema changes and Coren can do whatever magic is required to actually implement things on the instances. [17:42:00] andrewbogott: o/~ it's some kind of magic o/~ [17:42:08] (Sorry, was in a meeting with CT) [17:43:16] Coren: This is what I have, so far… https://wikitech-test.wmflabs.org/wiki/Special:NovaProject [17:43:33] Ugly but consistent [17:44:13] andrewbogott: Yeah, I'll do the instance implementation part thing as soon as I have confirmation of the schema. [17:45:00] andrewbogott: I can't seem to create a TEST Coren account on the test [17:45:31] Hm, what does it tell you? [17:46:00] There was either an authentication database error or you are not allowed to update your external account. [17:47:53] Coren: I just now created an acount successfully. So… maybe TEST Coren already exists? [17:48:50] There is no user by the name "TEST Coren" [17:49:11] (Trying to passwd reset) [17:50:02] Same with case variations. [17:50:29] Hm. [17:50:36] ... o_O [17:50:40] Um… you can log into nova-precise2 to debug, or just skip it for now :) [17:50:43] and creating an account just worked now. [17:50:49] Oh, well then! [17:51:08] No Nova credentials found for your account. (in a really BIG font) [17:51:34] You can discuss the font with Ryan; I'm not a big fan of that theme. [17:51:56] Let's see… for on thing you'll need to to run on two factor auth in order to do anything interesting. [17:52:00] :D [17:52:03] Also, I think I need to set a flag for you, lemme look. [17:52:17] Coren, you're using 'TEST Coren' as your username? [17:52:34] andrewbogott: Yep [17:52:38] we can switch the theme back if you'd like. it's for testing [17:52:51] No, it's pretty. Just a bit... unusual. [17:53:05] the font sizes need to be adjusted [17:53:14] and the whitespace is a little weird [17:53:36] I say. I think 72-odd points for "Access denied" looks a bit too much like MovieOS. :-) [17:53:41] heh [17:53:46] it's a h1 [17:53:50] *an [17:54:05] but yeah, the font sizes are huge [17:54:16] this has saner font sizes: https://wiki.openstack.org/wiki/Main_Page [17:54:18] it's the same skin [17:54:24] different css [17:54:36] Big fonts for the camera [17:54:36] I also adjusted its whitespace [17:54:55] for instance: https://wiki.openstack.org/wiki/Special:RecentChangesLinked/Main_Page [17:54:56] normal sized h1 ;) [17:55:11] It does. I do like the sober black bar. [17:55:12] still larger than mediawiki's defaults, though [17:55:20] yeah, I like the black theme [17:55:31] this lighter theme goes better with openstack's other apps. though [17:55:31] * andrewbogott still likes sidebars better than pulldowns [17:55:38] but can adjust [17:55:48] sidebars waste so much space :) [17:55:57] maybe if the sidebar was hidable [17:56:03] andrewbogott: I use my ipad a lot. Sidebars are a pain. [17:56:21]