[00:14:44] !log wikistats - updating mw versions of wmf wikis [00:14:46] Logged the message, Master [00:22:55] !log wikistats - adding wikivoyage to wikimedias_html table [00:22:56] Logged the message, Master [00:25:56] Krinkle|detached: Sorry I missed you; I'm around if I can make one for you. [00:31:16] Ryan_Lane: The last Puppet run was at Fri Mar 22 18:20:20 UTC 2013 (4690 minutes ago). Worrisome? [00:31:30] Krinkle|detached: (read backlog) ecmabot created [00:32:00] andrewbogott: just the service user [00:32:20] Coren: is puppet broken? [00:32:32] does it not run? [00:32:41] I don't know; I can make a manual run to check, I wanted to see with you first. [00:33:01] people often break puppet in labs by pushing a change into production [00:33:16] if it's broken it needs to be fixed, since that'll cause instance creation to fail [00:33:25] * Coren checking now [00:33:46] I didn't want to change the state if you wanted to check something first. [00:34:05] notice: Finished catalog run in 39.72 seconds [00:34:10] hm [00:34:18] I wonder if this is related to the cron change [00:34:23] are those running puppetmaster::self? [00:34:33] not -login [00:34:38] I have an instance just for that [00:34:42] ok [00:34:51] they should be running a cron now [00:34:54] and not an agent [00:35:28] instances running puppetmaster::self will likely continue running agents [00:35:29] * Coren wonders. [00:35:30] which is annoying [00:35:41] ok. gotta run [00:35:51] Hm. Did someone push the no-more-agent before they pushed the run-from-cron? [00:35:59] That would have been... unwise. :-) [00:36:11] no [00:36:14] at the same time [00:36:20] well, I did kill the agent everywhere [00:36:29] I'll keep an eye on it. Go. *wave* [00:36:36] * Ryan_Lane wave [00:38:15] Ryan_Lane: FYI for your backlog; the manual puppet run installs the cron entry. You may have killed the agents too fast. :-) [00:57:27] !log wikistats - fix sorting in wikimedias table [00:57:28] Logged the message, Master [01:34:27] @notify petan [01:34:27] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [01:41:26] Change on 12mediawiki a page OAuth/status was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=663715 edit summary: [+218] Pointer to the [[Auth systems]] page [02:06:11] !log wikistats - installing wikistats package 2.2 [02:06:13] Logged the message, Master [02:06:20] gone for real [08:41:11] !log bots petrb: creating -bnr4 to handle the queue backlog [08:41:13] Logged the message, Master [08:43:36] !log bots root: moving crontabs from bots-3 to bots-4 [08:43:37] Logged the message, Master [08:47:36] !log bots petrb: deleted bots-3 [08:47:37] Logged the message, Master [08:49:24] Ryan_Lane ping [08:49:45] Ryan_Lane, andrewbogott_afk, paravoid I can't create instances in bots :/ [08:51:55] mutante-away? [08:53:24] meh fixed... [08:55:04] hello [09:43:11] @search gr [09:43:11] Results (Found 5): newgrp, load, hexmode, -f, msys, [09:44:04] !botsgrid [09:44:39] !ping [09:44:39] @search debian [09:44:39] Results (Found 1): quilt, [09:44:41] pong [09:45:16] @search way [09:45:16] Results (Found 2): afk, del, [09:45:16] @search sge [09:45:16] @search grid [09:45:17] No results were found, remember, the bot is searching through content of keys and their names [09:45:18] No results were found, remember, the bot is searching through content of keys and their names [09:45:36] @search cable [09:45:36] Results (Found 1): gridbots, [09:45:46] meh [09:50:14] addshore ping [11:22:01] Hi, I would like to access bastion.wmflabs.org. I uploaded my pub key to gerrit. [11:22:47] I am approved to access right but I could not find name at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion [11:26:36] You can find my name at http://www.mediawiki.org/wiki/Developer_access/Archive [11:26:57] petan: pong [11:28:20] <^demon> ryuch: Added you to bastion. [11:29:31] <^demon> ryuch: You might still need to be added to the shell group, but I don't have access to do that. [11:30:36] Actually, I am in the shell group. I can see it on https://wikitech.wikimedia.org/wiki/Special:Preferences [11:35:20] [bz] (NEW - created by: Jasper Deng, priority: High - major) [Bug 39446] Spamming on en.wikipedia.beta.wmflabs.org - captcha not working? - https://bugzilla.wikimedia.org/show_bug.cgi?id=39446 [11:35:20] I think the missing of my name on https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion is the matter. [11:35:50] <^demon> You're on the list now tho :) [11:36:13] thanks for adding [11:36:33] still i am being denied. [11:38:43] [bz] (REOPENED - created by: Antoine "hashar" Musso, priority: High - major) [Bug 41104] glusterfs log files are not rotated - https://bugzilla.wikimedia.org/show_bug.cgi?id=41104 [11:45:01] Thank you, I made it. [11:46:36] I wonder what you did for me. anyway, thank you. no need to wonder! [12:19:05] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 45868] [OPS] let search indexer access the squid public ip - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 [12:33:37] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Low - enhancement) [Bug 45706] shell wrapper to connect to databases - https://bugzilla.wikimedia.org/show_bug.cgi?id=45706 [12:33:51] [bz] (NEW - created by: Chris McMahon, priority: High - normal) [Bug 46459] Search should work in beta UI - https://bugzilla.wikimedia.org/show_bug.cgi?id=46459 [12:34:05] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Low - normal) [Bug 46359] use XFS for varnish - https://bugzilla.wikimedia.org/show_bug.cgi?id=46359 [12:35:55] hmm petan bsql01 is going extrememly slowly again >.< [12:39:43] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 41135] on beta wmf-config/extension-list is a live hack - https://bugzilla.wikimedia.org/show_bug.cgi?id=41135 [12:40:26] [bz] (NEW - created by: Matthias Mullie, priority: High - enhancement) [Bug 42188] Allow for something other than master to be deployed - https://bugzilla.wikimedia.org/show_bug.cgi?id=42188 [12:42:15] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Lowest - enhancement) [Bug 41530] setup redis on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=41530 [12:42:16] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - minor) [Bug 45868] [OPS] [worked around] let search indexer access the squid public ip - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 [13:01:45] addshore hey [13:01:50] hey :) [13:02:03] addshore first: what about ur files, second why there are 2k instances of ur bot? [13:02:08] I had to create bnr4 to handle the load [13:02:31] petan: it was just a back log of processes as bsql01 slowed down [13:02:40] its back up to speed now and the backlog is gone :P [13:02:40] btw it's normal for 1 sql to be slow when 4 huge servers are flooding it [13:03:46] only 11 jobs now ;p [13:03:54] aha [13:03:58] maybe I could kill bnr4 then [13:05:07] yee :P [13:05:30] I may try and create a check that processes wont spawn if over 500 in the jobs already or something similar [13:08:02] addshore what about files [13:08:06] are they gone [13:08:09] yep :) [13:08:19] I found that after I deleted the files you wanted there were many others with different name [13:08:39] im going to make a cron to delete /home/addshore/wd.*.* every day :P [13:08:47] lol [13:08:51] submit it to grid :D [13:08:55] i will ;p [13:08:58] xD [13:51:54] Morning, Labs! [14:00:39] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Low - enhancement) [Bug 46104] reduce the number of wiki on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=46104 [14:31:15] !log deployment-prep Reduced the number of wikis on beta {{bug|46104}} {{gerrit|55889}} [14:31:28] !log deployment-prep refreshed wikiversions.cdb [14:54:19] petan: So… fixed? [14:54:35] andrewbogott fixed by deleting other [14:54:38] instances :P [14:54:46] I believe the problem was that we reached some limit [14:54:49] Ah, so you were just over quota. [14:54:54] but labsconsole was just saying "failed" [14:54:59] Let me know if you need me to adjust that at some point. [14:55:10] Yeah, labsconsole is basically the worst at relaying failure messages [14:55:19] I still /hope/ we will switch to tools project of Coren one day... but... [14:55:32] "hope"? [14:55:36] so I don't know if we need to raise quotas etc [14:55:54] andrewbogott: ui design issue? [14:56:12] Coren: well... I can't imagine to switch right my own bots now :) but hopefully some day I will [14:56:18] * my bots right now [14:56:38] which reminds me you installed mono? [14:56:45] I wanted to move one of my bots to tools [14:56:46] Long, long ago. [14:56:50] yes I totally forgot [14:56:52] sumanah: It is technically a UI problem, but getting detailed error messages from openstack is pretty complicated since the failures are asynch. [14:57:02] Did you get my pm? [14:57:21] So not so much a design problem as a 'not worth the months it would take to do right' problem :) [14:57:42] andrewbogott: ah, ok. Just slightly saying "there should be a bug report or something" - this sounds like something a volunteer could theoretically help with [14:57:54] or an intern [14:58:05] Yep, there's a bug for it. [14:58:20] can you cc me on it? [14:59:23] Yeah, because right now the OS extension has very Windows error messages. "Something has occured." [14:59:26] :-) [14:59:37] universal answer to all people who don't know why people prefer commercial microsoft's solutions over open source... there is no good documentation for most of open source libs :/ just compare .net and mono documentation... it's like 10 000 000 of pages and examples vs almost nothing [14:59:59] I switched to GTK and since then it's all so damn hard :/ [15:00:47] off to another meeting [15:04:09] [bz] (NEW - created by: silke.meyer, priority: Unprioritized - normal) [Bug 45483] Make instance creation failures more verbose - https://bugzilla.wikimedia.org/show_bug.cgi?id=45483 [15:04:46] petan: That's because you're not using the /right/ toolkit. Qt is very well documented. :-) [15:05:07] no don't tell me to switch again... [15:05:16] I just rewrote like 2000 lines of codes [15:05:39] Of course not; just travel back in time and don't have switched to GTK at all. :-) [15:05:59] I think GTK are better supported and true open source [15:06:05] QT are too much nokia [15:06:32] not that mono wasn't all about novel [15:07:06] actually I like QT too, but not sure if there is c# wrapper yet [15:07:44] "true open source"? How can you get more "true open source" than LGPL? [15:07:44] at least QT is not that ugly [15:07:54] of course you can... [15:07:59] compare mysql and maria [15:08:23] You're confusing "development model" with "source status" [15:08:32] both is open source, but one is controlled by some company and second is community maintained - anyone can join and work on it [15:08:36] Qt is definitely Cathedral. [15:09:01] I am not confusing it I am just mixing it all into one :) [15:09:29] Qt also has a clear design, and a well-defined development cycle. It's *possible* to succeed in true Bazar (to wit, Linux). GTK is proof that it can fail *hard*. [15:09:31] open source project to me is a project which is "open" in means that anyone can participate on its development not just read the code [15:09:47] petan: If you think you can to better, fork! :-) [15:10:14] but forking without merging has no point [15:10:48] if I fork big project into small local copy and add one minor feature, how other will benefit from that? [15:10:50] petan: If you can really do better then merging isn't a problem; you'd pull an egcs and become the one the old branch wants to merge /with/. :-) [15:10:51] nobody even notice [15:11:16] ok that's cool for big changes and ideas but not minor changes [15:11:49] even linux kernel is not what it used to be, these days it's all being developed by paid devs [15:11:52] GTK is a horrible mess of undocumented bad code /because/ of its development model. Bazzar isn't a panacea, and fails without a few benevolent dictators that kinda make it not-bazzar. :-) [15:12:46] I think that only reason why QT is better than GTK is that it's funded by nokia [15:12:58] lot of full time developers behind it [15:13:06] GTK is just open source [15:13:09] the old school one [15:13:12] petan: It spent a decade before that funding. It's better because it has a unified design, and a strong vision. [15:13:16] same difference as debian vs ubuntu [15:13:33] but before that funding it wasn't better than GTK :P [15:13:43] I didn't even hear about it before that [15:14:15] petan: Yes it was. The only reason you didn't hear about it is because of FSF politics. [15:14:50] I think if some company put few millions into development of GTK, just as nokia did with QT, it could become just as cool as QT is [15:15:01] what I am afraid of, is that once nokia stop funding QT it will die [15:15:09] while GTK will be still here [15:15:24] petan: It might; as long as they also put in strong control, development direction, and change management. I.e.: do things like Qt. :-) [15:15:49] petan: The difference isn't about funding, it's about direction. [15:16:00] what do you mean by strong control - restricting people from being able to contribute? [15:16:19] well, I believe it's all about funding :P [15:16:33] petan: You're confusing "quality control" not "restricting". Quality control /means/ having to restrict. [15:16:45] if you just changed the direction and didn't hire tons of full time devs, I don't believe you would see any improvement [15:16:46] What's what control /means/. :-) [15:17:50] petan: Just you /try/ to make anything but the most trivial of patches merge in GCC. It'll take a lot of time before you are "allowed" to. That's a normal evolution once something is mature and depended upon. [15:18:02] by making development process complicated you will discourage many developers from participating [15:18:12] petan: And yes, sometimes a shakeup is necessary with a fork. But that's uncommon. [15:18:46] petan: Yes, and you will increase the quality of the product. The trick is trying to find the right balance. [15:19:11] petan: Good crowdsourcing also means having to accept that not everyone who /wants/ to contribute have the skill to do so usefully. [15:19:17] GTK is bloated by template obsfucation [15:20:32] Hi, again. I would like to be a member of bot project. I am running RFC bot for Korean Wikipedia. I want to do some work to migrate the bot from toolserver. [15:20:44] Coren ok but there might be potentially good contributors who just lack the skill, instead of mentoring them, this philosophy is discouraging them [15:20:52] Coren look at wikipedia that is great example [15:21:21] almost anyone can edit - if it was restricted to certified editors who are known for being good encyclopedia writers - it wouldn't be so good as it is [15:21:27] despite the quality might be better [15:21:52] wikipedia is not considered as reliable as some comercial encyclopedias, but look at yourself [15:22:09] if you need to find something, you will always look to wikipedia because it contains more stuff [15:22:16] petan: True Open Source(tm) isn't egalitarian, it's a merit-based aristocracy. [15:22:52] ryuch hi [15:22:59] ryuch what is your name on labs? [15:23:03] @labs-user Ryuch [15:23:04] Ryuch is member of 1 projects: Bastion, [15:23:08] aha here we go :) [15:23:09] hi, petan. ryuch [15:23:37] petan: Same thing with Wikipedia, really. The barrier of entry is low, but not absent. And, besides, the requirements are different. Nobody can break the article on "Bears" by making a bad edit to a small biography or an article about a village. [15:23:53] Heya, Ryan_Lane [15:24:02] ryuch. (stupid autocomplete) [15:24:19] i know Ryan is famous here [15:24:42] !log bots Ryuch to project :o [15:24:50] ryuch you are there [15:24:55] btw we are missing a bot in here [15:24:58] thanks, petan. [15:25:21] Coren on wikipedia barrier of entry is absent... you just need an internet :) [15:26:04] petan: My block button and checkuser say different; and there is no difference between a patch being rejected and an edit reverted. [15:26:30] the reason why wikipedia works is that it is so open and that the community is so large... there are many more people who are "good" either in editing and in making sane changes than number of people who are either vandals or newbies [15:26:43] so skilled editors will always fix and mentor new [15:27:20] that makes no need to restrict editing only to some "elity" [15:27:33] petan: This isn't the time, nor the place, for a philosophical discussion about open governance structures. Software (and infrastructure) have different needs for quality control. :-) [15:27:43] mhm [15:27:51] doing minor change in a wp: article is easy, but nowadays many things require to break through half a dozen of bureaucracy layer [15:28:04] @labs-user ryuch [15:28:04] phe: yes and that suck [15:28:12] phe: That's the normal evolution of any project as it matures. [15:28:18] ryuch no extra space :P [15:28:21] @labs-user Ryuch [15:28:21] Ryuch is member of 2 projects: Bastion, Bots, [15:28:24] See? Philosophical debate again. :-) [15:28:32] great! [15:28:34] ryuch you had @labs-user [15:28:42] and worst many other things require to break through half a dozen of template layer :) [15:29:21] phe: That is because, in the end, Mediawiki is... lacking. It's getting better though. :-) [15:29:30] btw Coren & phe if you just wanted to tell me that choosing GTK was a bad choice, that information really came in a very bad time :P and you wouldn't make my day [15:29:48] petan: It's not a *bad* choice. There are tradeoffs. [15:29:57] I just finally started to feel like I am moving somewhere good [15:30:14] at least it works far better on linux than winforms [15:30:17] petan: But you'd win by being familiar with both anyways. :-P [15:30:37] I can become familiar with QT but I don't want to rewrite that application again [15:31:02] petan: Just write the next one with Qt then decide for the future. Rewriting is only fun up to a point. :-) [15:31:30] btw is there a good layout editor for QT? cross platform? [15:31:33] something like glade [15:31:58] petan: Qt designer. Part of the dev packages. [15:32:41] http://qt-project.org/doc/qt-4.8/designer-manual.html [15:32:41] Coren: here we go http://stackoverflow.com/questions/3023362/best-c-sharp-bindings-for-qt [15:32:49] qt designer and layout is a pain to use, I pass most of my time to break things already do correctly ^^ [15:32:49] what a surprise :P [15:33:04] project which isn't so "open" can't really have so powerfull community as gtk has [15:33:39] petan: failed to create? [15:33:44] petan: You're confusing cause and effect. There aren't very many Qt bindings for C# because there aren't many Qt users who willing use C#. :-) [15:33:45] there are just no people who are actually working on it, which doesn't surprise me given the amount of bureaucracy insinde of QT [15:33:48] i'd imagine you're hitting quotas again [15:34:02] willingly* [15:34:03] Ryan_Lane yes, solved by deleting some instances [15:34:14] Ryan_Lane problem is that error message is just "Fail" [15:34:24] so I don't know if I am hitting quotas or there is something wrong [15:34:41] well, seeing as that it's only ever really been quotas.... [15:35:25] Ryan_Lane: Err, actually, I can't create instances either. [15:35:28] Ryan_Lane ok then maybe change "Failed" to "Unable to create new instance because you exceeded the limits" or something like that :P [15:35:38] Ryan_Lane: And I got way may fewer instances than bots. [15:35:41] Coren: you are probably also hitting quotas [15:35:46] Ryan_Lane: Are we out of some resource? [15:35:51] Coren: it's based on a number of factors [15:35:56] Coren we are out of wires :P [15:36:16] Oh noes! We're out of electrons!!1! [15:36:20] Coren: it's based on cores, memory, instances, storage, etc, etc [15:37:06] Coren btw if it was about popularity of c# (which is huge actually, so don't think that is problem) why GTK has solid wrappers and QT doesn't? maybe because GTK community is larger and more friendly and open... but let's get back to labs things [15:38:30] !wmflabs is http://i215.photobucket.com/albums/cc152/estaquente/funny_pictures/e744f91c29ec99f0e662c9177946c627.jpg [15:38:30] Key was added [15:38:33] :P [15:38:39] this is a picture from datacenter [15:38:40] petan: No, I'm talking about (C# coders) ∩ (Qt users) [15:39:26] Coren I believe there are some hence the questions on stack :P [15:39:40] the picture is better than my desktop [15:39:40] Some, but I find most Qt coders are C++ coders. [15:39:56] yes because QT only works with c++ :/ [15:39:59] Coren: ok. I've tripled tools' quota [15:40:09] Ryan_Lane: Danke. [15:40:10] if I was making GUI in c++ I would use QT as well... [15:40:14] MOAR POWER!!! [16:03:04] Ryan_Lane: So if I were to set up an Apache instance on a labs machine, should I do that via Puppet, preferably, or would apt-get be an acceptable way of accomplishing it? [16:03:58] marktraceur: you can use apt-get. there's no reason you're forced to use puppet. [16:04:07] Ryan_Lane: I mean, in a best-practices sense [16:04:17] well, then I'd recommend using puppet ;) [16:04:20] Cool. [16:04:32] Ryan_Lane: The next question is where can I read documentation on how to do that? :P [16:04:36] because if you ever want to move to a new instance it's easier [16:05:06] marktraceur: https://wikitech.wikimedia.org/wiki/Help:Instances#Puppet_information [16:06:14] On it, thanks. [16:08:44] yw [16:12:17] New patchset: coren; "Add my key (Marc-Andre Pelletier)" [labs/private] (master) - https://gerrit.wikimedia.org/r/55898 [16:12:23] Ryan_Lane: So now I have these service users which are in kind of a grey area. There are places in the code (e.g. the sudoer stuff) which doesn't see the service users because OpenStackNovaUser doesn't see them. [16:12:46] Do you have any instinct about whether I should extend OpenstackNovaUser to see them, or just add special case code elsewhere? [16:12:53] extend, I'd say [16:13:28] Hm… of course then NovaUser will have to keep track of project context somehow... [16:13:41] Well, I'll try that, at least. Will see if it backfires. [16:13:49] Is it /relevant/ to see project-local users in those contexts? [16:14:10] * Coren would have thought not. [16:14:21] Coren, the problem I'm looking at right this minute is things like this: https://nova-precise2.pmtpa.wmflabs/w/index.php?title=Special:NovaSudoer&action=modify&sudoername=local-servotastic-chmod&project=sudotest [16:14:44] The whole point of that sudo policy is to allow the service user to do things. But the sudoer gui doesn't know about the service user. [16:15:05] Oh. Oh, yeah, that does make sense. [16:15:09] I'm not sure yet; that might be the one and only place where it matters though. [16:15:32] But that might actually be worth a special case; I'd think that we want to give sudo rights to human users in general. [16:15:41] There are several places where the code says "Tell me about this user" and passes in the service user… and it just gets back 'null' [16:15:56] andrewbogott: well, they are prefixed. maybe skip lookup on usernames with a prefix [16:16:05] and list them as local-blah (service user) [16:16:51] Yeah, maybe OpenstackNovaUser doesn't have to actually know anything about them and can just generate a user record on the fly based on the prefix. [16:16:55] Hm... [16:17:02] * andrewbogott will try that after lunch [16:17:29] Ryan_Lane, meanwhile, I'd appreciate you having a look at the ldap code in https://gerrit.wikimedia.org/r/#/c/55125/ [16:18:32] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: High - normal) [Bug 45084] autoupdate the databases! - https://bugzilla.wikimedia.org/show_bug.cgi?id=45084 [16:19:52] looking [16:38:58] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/55898 [16:39:00] Coren: ^^ [16:39:08] it'll update when puppet runs [16:54:00] Noted, and worky. [17:16:30] Ryan_Lane: So that I can fix the problem when it happens now; my new instances seem unable to mount volumes (which kills puppet because of addusers in it). What is your usual fix? [17:16:57] what do the logs say? [17:17:15] I'd imagine the daemon that manages access hasn't added the new instances yet [17:17:26] what's the instance names, I can check [17:17:32] tools-exec-02 is one [17:17:59] Nothing in the log: the /usr/sbin/glusterfs --volfile-id=/tools-home just sits there [17:18:11] not in the list [17:18:53] "the list"? [17:19:28] Ah; on the gluster itself. Gotcha. [17:19:33] It's been a while, though. [17:20:06] yeah. the list is weird, too [17:20:32] "wierd" doesn't sound like a desirable quality in that context. [17:22:22] ,10.0.0.41,10.0.0.42,10.0.0.43,10.0.0.44,10.4.0.128,10.4.0.156,10.4.0.220,10.4.0.240,10.4.0.8,10.4.1.54,10.4.1.66,10.4.1.89 [17:22:29] notice the , with nothing before it [17:23:26] Hm. I'd not wager on how it copes with an empty string in an access list. [17:23:41] doesn't seem like it matters [17:23:53] but, it's missing the IP [17:24:32] of the instances [17:25:03] it hasn't updated in a couple of hours [17:25:38] Should have 10.4.1.{65,66} there [17:25:58] 66 is there [17:26:19] I see. I added both pretty much at the same time (i.e. within 60 secs) [17:26:48] I'm running the daemon manually to see if there's any tracebacks [17:27:16] I wonder if you hit a race condition I haven't seen in ages [17:27:38] hahaha [17:27:40] indeed you did [17:27:50] dc: i-0000064c [17:27:51] aRecord: [17:28:00] it's missing an IP address in ldap [17:29:21] Huh. [17:29:34] So it added it to the dhcpd, but not ldap. How evil. [17:29:54] dns is managed by OpenStackManager for now [17:30:14] a job gets injected into the job queue when an instance is created [17:30:28] the job gets run later, which adds the ip [17:30:49] it can't add it immediately, because instances are created, then given an IP address after they are scheduled [17:31:13] we'll be switching to using moniker (which is an openstack project likely to enter incubation soon) [17:31:52] moniker actually does DNS updates through DNS right? [17:33:06] we'll continue using an ldap backend [17:33:20] or maybe we'll switch to a new backend when we switch production dns as well [17:33:40] I'm not amazingly happy with pdns's ldap backend [17:34:27] Ryan_Lane: I note the mysterious appearance of /home. Fix't? [17:34:34] yep [17:35:03] ugh. upstart is such a giant piece of shit [17:35:19] if you run restart and something isn't running, it just fails [17:35:40] rather than how it worked with init, where it'll fail the stop, but continue to do the start [17:35:54] which is the sane approach [18:02:43] Ryan_Lane: you mean the software written by the guy who's personal blog is reposting his "No more Ubuntu machines" post to planet.ubuntu.com repeatedly isn't a totally sane peice of linux plumbling? [18:03:03] heh [18:03:21] (who now works for Google, I believe) [18:16:08] !log tools Doubled the size of the compute grid! (added tools-exec-02 to the grid) [18:16:17] andrewbogott_afk: ok. did a review [18:16:26] lots of code in this change :) [18:16:26] * Coren looks around for the bot. [18:16:32] hm [18:16:35] bot dead? [18:16:55] Comatose at best. [18:17:04] he wasn't in the room at all [18:17:12] !log tools Doubled the size of the compute grid! (added tools-exec-02 to the grid) [18:17:14] Logged the message, Master [18:17:16] labs-morebots: what's your deal? [18:17:17] I am a logbot running on i-0000015e. [18:17:17] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [18:17:17] To log a message, type !log . [18:20:58] Now to see if I can remember to to setup a shadow master after having done it once, ~6 years ago. :-) [18:42:56] Coren: shadow master? [18:43:32] It's the server that directs the execution ninjas behind the scene. [18:43:35] :-) [18:43:49] SGI gridmaster failover server. [18:44:12] That's their terminology. [18:44:19] ah [18:44:20] ok [19:21:51] Ryan_Lane: OK, so, I create a 'group set' that has ou=group,… and that ou has members, each of which is the dn of a service group. [19:22:08] Are you saying that that's unnecessary and the group set doesn't actually need a member list? [19:38:00] Ryan_Lane, back? [19:39:50] * Damianz eyes Ryan_Lane [19:56:22] [bz] (NEW - created by: silke.meyer, priority: Unprioritized - normal) [Bug 45483] Make instance creation failures more verbose - https://bugzilla.wikimedia.org/show_bug.cgi?id=45483 [20:17:29] <^demon> None of my instances are showing up on Special:NovaInstance. [20:30:50] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: High - normal) [Bug 45084] autoupdate the databases! - https://bugzilla.wikimedia.org/show_bug.cgi?id=45084 [21:20:21] ... [21:20:36] labsconsole is dead again, no instance showing up :-] [21:20:45] bed time I guess [21:20:54] <^demon> Yeah, I complained an hour ago. [21:20:57] <^demon> Nobody said anything. [21:23:59]