[00:00:15] Krinkle: U can haz curlz! [00:05:06] Coren: https://svn.toolserver.org/svnroot/puppet/trunk/puppet/modules/software/manifests/misc_user.pp [00:05:15] Something to dig through perhaps [00:05:27] there's lots of other puppet manifests but this looks like the main one for the user land [00:06:10] https://svn.toolserver.org/svnroot/puppet/trunk/puppet/modules/software/manifests/php.pp [00:06:17] (for php* modules* [00:06:25] php5-mysql php5-cli php5-sqlite [00:06:29] those are definitely required [00:07:04] Coren: Where in wmf puppet should I add these? I can hand-pick a selection that I belief are common enough beyond doubt [00:07:26] Or perhaps just add them on a need base? [00:09:16] Krinkle: Hypothetically, in production/puppet in the tools::run_env class (in the tools module). But, atm, I'm doing the manual dance while we're in setup mode; I don't have +2 so the turnaround is too long for interactive point patches. [00:09:29] Hence my Notepad for "stuff that will have to go in pupet" :-) [00:10:03] So I do as-needed and will commit the bunch at once sometime in the next two weeks as more tools move over. [00:10:24] The other three php-* you mentionned were already there, though. [00:12:18] Coren: Ehm, you broke something [00:12:34] o_O [00:12:38] Coren: https://gist.github.com/Krinkle/6116129bd6bc42a5b015 [00:13:00] Ah. You ran out of memory. [00:13:08] php is... not helpful in that case. :-) [00:13:21] *I* ran out of memory? The process didn't even start yet. [00:13:32] I get the same error when opening a php prompt. [00:13:37] Something happened at compile time I guess? [00:13:43] (when you installed the extension) [00:13:45] job 133 is yours. [00:14:09] Well, so /you/ didn't run out of memory. Yours did. :-) [00:14:41] How can that be [00:14:44] I don't have anything running [00:14:52] The process is killed within a millisecond of it starting [00:15:04] job-ID prior name user state submit/start at queue slots ja-task-ID [00:15:05] ----------------------------------------------------------------------------------------------------------------- [00:15:05] 134 0.25000 dbbot-wm local-wmfDbB r 03/23/2013 00:14:16 continuous@tools-exec-01.pmtpa 1 [00:15:07] I jstop'ed it and cleared the log files [00:15:09] re-submitted [00:15:17] and instantly 10 of those lines in .err again [00:15:28] Yeah, it's running out of memory as it tries to map the php modules. [00:15:52] 256M as default might be overly conservative. [00:16:21] You can make sure by starting it with -m 512m for instance [00:16:23] Why do you think it is a memory problem? [00:16:52] I've seen it before; that specific way of breaking is due to the way php dlopens() its stuff. [00:17:09] It doesn't actually check that it worked for real. [00:19:16] http://www.litespeedtech.com/support/forum/showthread.php?t=6183 [00:19:26] Looks like it requires memory allocation [00:19:32] not in use, but just allocation [00:19:36] php in general, not my script [00:19:52] Once certain modules are installed it automatically wants to reserve more memory [00:20:00] Yes, that's what I meant. php is the one that deals badly with memory allocation. [00:22:26] Coren: Where is 256 set? Should you increate memory limit in php.ini ? [00:22:39] No, it's actualy memory use, not internal php use. [00:22:49] You can jstart with -m 512m if you want to try [00:24:37] I'll have to bump default memory allocation up if just adding php modules increases its footprint. [00:24:43] Coren: Doesn't work, gives me usage help [00:24:51] jstart -N dbbot-wm -mem 512m php apps/ts-krinkle-Kribo/Init.php [00:24:54] oh, sorry. -mem [00:24:59] I know :) [00:25:01] * Coren tries [00:26:37] Oh. Silly me. I broke something with my previous fix. [00:27:35] hacky hax hax :) [00:28:32] fix't [00:28:51] * Coren tried all the options to boot. [00:29:23] Coren: Yep, the bot does start now [00:29:27] Odd [00:29:55] I can't look on gangla (keep falling away, no idea), but I guess you know [00:30:06] No, it's just that PHP is an ill-omened piece of refuse. It's been written presuming infinite memory and diskspace. [00:30:10] how much power is there allocated for one job runner at the moment (cpu/memory) [00:30:39] Coren: Yes, doesn't the cloud have infinite space? But daddy said so! [00:30:58] Memory is 256M by default, but no CPU allocation is made; you get at least 1/4 cpu since there are 32 slots over 8 cpu. [00:31:10] sorry, I meant the actual vm its on :) [00:31:38] Ah, that's in nova as well, don't need ganglia [00:31:39] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000604 [00:31:40] great [00:31:40] 8G / 4 cores [00:32:59] But I can simply add more at need, which is one of the more useful reasons of having a grid to begin with. :-) [00:33:06] yeah [00:34:08] Next week I'm adding two more and the shadow master now that we're in "almost-prod" [00:34:32] brb. Tea [00:38:01] back [00:39:03] I like the SGE terminology for that, at least. "Shadow master" sounds like the evil overlord dispatching his exec ninja minions around. :-) [00:39:47] uh.. what just happened? [00:40:03] Coren: My bot just died, no .err or .out message. And gone from qstat [00:40:20] No .err? [00:40:28] Nothing new in .err [00:40:52] And you had started it with jstart, yes? [00:40:53] [01:38] dbbot-wm has left IRC (Read error: Connection reset by peer) [00:40:55] Yep [00:41:04] Is there a case where it will exit with a return value of 0? [00:41:29] * Coren checks on logs. [00:42:00] It happened right after I made it update something (command @externals update, it pulls the latest wmf-config for analysis) [00:42:09] which executes a php command in the background [00:42:16] it worked fine a few minutes earlier [00:43:27] I'm trying one more time (just did jstart [00:44:52] Coren: Yeah, it's doing it again, consistently. It responds fine until I give it that command. [00:44:52] I saw it die. [00:45:05] * Coren chuckles. [00:45:09] It says " Running ./maintenance/updateExternals.php in the background.." and then it just goes away. [00:45:17] You're invoking another php [00:45:18] Nothing from there on in .err or .out [00:45:30] Can I look at your source? [00:45:59] The IRC framework is here: https://github.com/Krinkle/ts-krinkle-Kribo but I wouldn't bother looking there as that's running fine [00:46:25] https://github.com/Krinkle/ts-krinkle-wmfDbBot/blob/master/inc/Commands.php#L248 [00:46:36] the bot command "@externals update" invokes purgeExternals() [00:46:48] Owait. You're running /another/ php cli. That basically doubles your memory usage! [00:47:08] The limit isn't per process, it's per /job/ [00:47:10] But shouldn't it then say something in .err ? [00:47:14] Sure, I understand. [00:47:46] Depends on how your calling code deals with the external process going boom. Lemme look at the php code for passthru. [00:48:07] also it also dissapeared from qstat [00:48:13] interesting [00:48:24] I have error reporting on highest (everything is logged) [00:48:34] And yet nothing, not even a notice. [00:50:46] Oh for Larry's sake! No wonder PHP deals badly with lack of memory. [00:50:58] I raised it to 1024 to see if it would survive. [00:51:07] Looks like it it. Unfortunately [00:51:14] Coren: It's even worse? Tell me. [00:51:23] Double checking. [00:52:23] I need to test this. This is too funny for words. [00:54:38] Yeah; the first php (the one that calls passthru()) aborts if it runs out of memory _trying to construct the error message_ [00:54:42] https://github.com/php/php-src/blob/master/ext/standard/exec.c#L226 [00:54:57] Coren: XD [00:55:15] That explains the lack of a php error [00:56:00] And running out of memory is the one thing the keep-going script can't recover from because that kills the whole process tree. [00:56:03] * Coren ponders. [00:56:23] I could trap the signal; that'd restart the bot... [00:56:42] but then, it'd just go back down again next time it runs out of memory. [00:56:42] Coren: But ahm.. how is memory limit for jsub enforced? [00:56:49] It's a hard ulimit [00:57:08] does "it" (the grid wrapper) know about it? or is it pass to some other library that enforces it in linux? [00:57:18] Meaning, could you make it write to .err? [00:57:24] when it kills it [00:57:34] (explaining the reason) [00:57:40] SGE sets the actual ulimit; that's enforced by the OS. The OS won't /kill/ the processes, they'll just be unable to allocate a byte more. [00:57:50] Ah, I see. [00:58:08] Including, if your program is dumb enough, the buffer in which you're trying to write the error message telling that malloc() failed. :-) [00:58:29] By "your program", in this case, I mean "PHP" [00:58:32] So if it quits, it's because the program itself can't coop, not because jsub or the os kills it. [00:59:09] Right; I think that php tries to dereference malloc()'ed memory without even checking if malloc worked. [00:59:27] * Coren ponders. [00:59:41] 1G might be overkill, mind you. Did you test with the external script? [00:59:41] * Krinkle thinks of melody pond. [00:59:49] I did [01:00:08] If you stop your bot, you can use qacct to find out how much memory it actually ended up using. [01:00:08] And even gave the irc bot the command several times to see if there is any residu that builds up [01:00:11] but it's fine with 1g [01:00:17] but its too much really [01:00:23] it works, but I don't want it to work :P [01:00:27] Heh. [01:00:37] Virtual mem is cheap; reliability isn't. [01:00:57] Coren: Hm.. wait, this I gotta see. [01:01:01] Let's just say that php is not memory friendly. [01:01:24] How do I use qacct? [01:01:52] I get /var/lib/gridengine/default/common/accounting: No such file or directory [01:01:58] Oh, wait, that's true. Stoopid SGE. I need to do it for you; it doesn't give unprivileged users the right to query the gridmaster unless it's your login box. [01:02:01] qacct -j ddbot-wm [01:02:08] OK [01:02:39] What was your last job number? [01:03:08] 138 [01:03:44] Quiting it now [01:03:46] maxvmem 554.055M [01:04:09] Wait, that's 137 [01:04:49] maxvmem=568.680M [01:05:01] Oh, you can actually qstat though [01:05:07] qstat -j 138 should tell you [01:05:52] So yeah, it peaks just a midge over the 512M limit. :-) [01:05:53] usage 1: cpu=00:00:01, mem=0.08922 GBs, io=0.00974, vmem=220.328M, maxvmem=568.680M [01:05:55] indeed [01:06:27] 'qstat -j dbbot-wm' works, too. That's nice. [01:06:55] Huh. I didn't even know you could use job names. :-P [01:08:26] aight, nuff for today. Thanks for everything. Now I got enough details to migrate everything else I got going on at toolserver (eventually) [01:08:46] Might ping you in a few days/weeks for additional packages. [01:09:01] Don't hesitate. I'm idling on here 99% of the time. [01:09:31] Of file a bz which makes my phone go bzzz1 [01:09:33] :-) [01:09:43] OK, I'll bite. What's with the 1%? I'm idling 100% (not awake 100%, but idling/bouncy) [01:10:10] I'm having trouble with my UPS lately. :-) [01:10:27] I see. uptime [01:10:37] Much to my annoyance, that. [01:11:07] Looks like my bouncer is at 12 days now. Last downtime was a scheduled reboot that only took a few minutes. [01:11:11] marc@mordor:~$ uptime [01:11:11] 21:10:55 up 1 day, 11:10, 3 users, load average: 0.05, 0.10, 0.14 [01:11:11] Small uptimes make me sad. :-( [01:11:53] A point of note; that box sits on a desk so that one cannot simply walk into it. [01:12:00] connected from the bouncer is 12 days. Though uptime of the server is slightly better: 8w 3d 7h 16m 48s [01:12:24] had to restart irc, or freenode freaking out [01:12:43] * Coren bahs. Nobody *ever* gets that joke. [01:13:09] Sorry, I.. I didn't. [01:13:20] [21:11:52] A point of note; that box sits on a desk so that one cannot simply walk into it. [01:13:31] [21:11:10] marc@mordor:~$ uptime [01:13:44] Ah! [01:14:10] and in the darkness bind them! [01:14:30] One day, ONE day I shall get someone who understands it and is, at least, mildly amused. I live in hope. :-) [01:14:41] Coren: LOTR fan? [01:14:59] LOTR complete otaku is more like it, I suppose. :-) [01:15:23] Coren: I got a nice did-you-know for you in that case [01:16:03] The poster for the first movie was designed at my dads company [01:16:11] He's a graphic designer. [01:16:17] The PJ adaptation? [01:16:25] Hanging proudly on my wall here. [01:16:33] The one where Frodo looks down on this hand with the ring. [01:16:59] Nicely collectible if you have the marquee vinyl [01:17:50] http://www.filmonpaper.com/site/media/2011/03/LOTR_Trilogy_onesheet_Frodolookingup-1-500x725.jpg [01:17:54] that one [01:18:38] They hired like a dozen agencies to come up with a design, and they picked this one. [01:18:48] Ah, one the the teasers. Nice and moody [01:19:10] Yeah, it's the teaser from before part 1 [01:19:25] Haven't seen the hobbit yet (read the book, of course). have you? [01:19:58] Yes, enjoyable given that they could take a great deal more liberties with padding. [01:20:32] oh man, people screw that up so often. It's unbearable. [01:20:37] I'm surprised it was so well recived, tbh, because the pacing is downright cerebral and character driven; most USian moviegoers can't stand that. [01:21:28] I'm one of those that enjoyed the PJ adaptation despite some of the more... dubious choices he made in adapting the source material. [01:21:54] I have high excectations given the trilogy. But then again I also know that this one seems to be from another nest that isn't as well funded / capable [01:22:06] Most I can see the rationale for, even when I disagree. Some were grating, but I was willing to overlook them given the visuals. :-) [01:22:27] At least they got the same actors. [01:22:36] Krinkle: They do have the advantage that the source material is much thinner and smaller in scope; this does provide enough elbow room. [01:23:02] What I appreciated is the fact that they did a good job with tone. [01:23:04] I was surprised to hear it is a film series though [01:23:49] Krinkle: Gives them room to expand. They actually delve deeper in the mythos and expose stuff from the Elder days that don't get discussed directly in the Hobbit [01:23:55] the hobbit did seem thicker than any of the individual lotr books (not sure), but 3 parts? [01:24:07] I mean, after all, the Hobbit was retconned to be in ME after all. :-) [01:24:15] I guess they'll extend beyond the book then. [01:24:37] which is tough, since it plays in the time before lotr [01:24:42] timeline* [01:24:45] Around the book is more like it. They flesh out a lot that is left unsaid, and give nice background on the Elder days. [01:24:50] so they can't get off too far. [01:25:59] Alrighty, I havent' seen the film yet, so enough without spoilers (or we get River Song into the mix) [01:26:11] They delve in the past rather, which is nice, because it's not like they could possibly make a movie around the Silmarillion. :-) [01:27:28] Well, they could, but it would make Shindler's List look like a short action flick by comparison. :-) [01:27:36] lol [01:28:33] Though I didn't learn them, I also have some of the books about the environment and the languages [01:28:42] maps etc. [01:28:45] quite impressive [01:28:56] So it's a safe bet to presume you play LOTRO then? :-) [01:28:58] one could write a 100 more stories, it's a complete world. [01:29:07] 'play' ? [01:29:11] * Krinkle googles LOTRO [01:29:14] LOTR*O* [01:29:27] OMG. You don't know about LOTRO? [01:29:28] No, I don't play any online games [01:29:40] Just heard about LOTRO for the first time. [01:29:59] You *have* to. They /suceeded/ in making Middle Earth look and feel credible. [01:30:16] Lemme tease you with an anecdote. [01:30:23] Maybe after they get their shit together: LOTRO [01:30:31] Maybe after they get their shit together: http://cl.ly/image/3D371v2S2c3V [01:30:47] service unavailable xD [01:30:51] o_O [01:31:01] First time I see this. Talk about bad timing. [01:31:08] So, brief anecdote: [01:31:13] Anyway, go ahead. [01:31:41] First character I create, a hobbit. Same old idea; quest givers, do quests all that jazz. You start in Mitchel Delving. [01:31:50] * Krinkle is reading http://www.lotro.com/en now - it seems back up. [01:31:54] I get this quest, having to go to the Party Tree. [01:32:06] ok [01:32:36] The environment is so well made, and so consistent, that I actually could orient myself and find Bag End, and the party field, just by my mental maps and images of how things /ought/ to look like. [01:32:46] They are basing on the books, not the movies. [01:33:17] yeah, they cheated quite a bit in the movies. Especially in part 2 and 3. [01:33:54] And there is the "main" storyline you play through that is actually (a) consistent with canon (b) doesn't interfere with the Fellowship (c) but important in the world and interacts with the fellowship in ways that make sense! [01:34:41] It's a masterpiece, IMO, and I'm *very* picky [01:35:02] If you go for the trial, tell me! I'll be happy to guide you along on the Meneldor server. :-) [01:35:22] * Coren has a Lifetime subscription. :-) [01:35:36] Those no longer exist, sadly. [01:41:19] Coren: Bookmarked. Thanks! [01:41:34] Going to close up now, need to get up early tomorrow and almost 3am here. [01:41:38] cya [01:41:39] * Coren waves. [05:52:09] yargh who broke ganglia [12:01:51] addshore ping [12:02:03] I deleted the files you wanted but there is still a lot of them [12:02:09] of different name [12:02:27] STDIN.o300 [12:02:28] submit.e298 [12:02:29] submit.o298 [12:02:30] test.sh.e71949 [12:02:31] test.sh.o71949 [12:02:32] wd.dbstat.e100028 [19:24:39] addshore .. [19:29:28] wmfDbBot is a monster hog. In 15 hours it managed to eat up almost two whole seconds of CPU time! :-) [19:38:01] only 2? meh [19:56:14] Not related to Labs: What is the MySQL complement to "CONVERT(? USING utf8)", i. e., given a UTF8 string, what gives the VARBINARY presentation used by Wikimedia? [20:10:23] * jeremyb_ is sitting next to marktraceur [20:10:26] :-) [20:10:59] scfc_de: isn't the wikimedia string just utf8? [20:11:19] (but mysql's utf is somehow broken. so encoding/decoding is done in php) [20:11:27] * marktraceur waves [20:11:29] ask domas/binasher [20:11:47] The most important conclusion of LibrePlanet so far: Let's not get Congress to use Gerrit. [20:12:00] s/congress/anyone/ [20:12:04] s/congress/anyone we like/ [20:12:09] s/congress/not even our enemies/ [20:12:22] jeremyb_: Yep, but as type VARBINARY. Apparently, just "switching sides" works: "SELECT * FROM page WHERE page_namespace = 0 AND page_title = CONVERT('Ä' USING utf8);" [20:12:39] idk [20:17:44] marktraceur hey [20:18:12] petan: Howdy! What's up? [20:18:13] marktraceur could you insert me to etherpad project? I wanted to checkout how is it installed and help out with maintenance [20:18:17] Sure sure. [20:18:21] name Petrb [20:18:24] thanks :) [20:18:58] I was trying to install it on my own box but kind of failed [20:19:09] petan: Sorry to hear that :( [20:19:15] not really that I would failed but I didn't have time to finish it :D [20:19:19] petan: Added [20:19:27] because the ubuntu version doesn't have proper version of libraries [20:19:40] did you compile it yourself or used some packaged version [23:13:58] https://wikitech.wikimedia.org/w/index.php?title=Special:UserLogin&type=signup should probably link to the terms of use