[00:01:04] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 36748] [OPS] syslog::server (in test) unusuable - https://bugzilla.wikimedia.org/show_bug.cgi?id=36748 [00:28:53] This is not good, is it? v [00:28:54] http://ganglia.wmflabs.org/latest/graph.php?h=cvn-app1&r=week&g=mem_report&c=cvn [00:29:01] I suspect one of the cvn bots has a memory leak :( [00:29:20] when various vms dead earlier this week, it was saved by the bell [00:36:02] Krinkle: yeah, definitely looks like it [00:36:31] should be easy enough to figure out which one [00:47:02] !log deployment-prep Trying to import enwiki database on the lucene search deployment-searchidx01 : sudo -u lsearch /a/search/lucene.jobs.sh import-db enwiki [00:47:05] Logged the message, Master [00:47:15] xyzram: seems I have imported enwiki in the lucene search indexer \O/ [00:48:55] xyzram: https://wikitech.wikimedia.org/wiki/Deployment/Search :-] [00:59:49] hashar: Great. [01:12:18] I need to fill bugs [01:17:55] xyzram: I will fill bug later on :-] [01:20:09] [bz] (NEW - created by: Krinkle, priority: Lowest - enhancement) [Bug 34250] [beta project] Set up search (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=34250 [01:22:20] ? [01:22:30] That was 11 months ago [01:22:51] its status is NEW but it isn't a newly created bug [01:38:12] Did Bastion's SSH host key change? [01:39:01] FastLizard4: no [01:39:06] is it showing as different? [01:39:17] Yes [01:39:29] Or, at least it was on my laptop [01:39:31] Shows the same for me. [01:39:38] Trying it from my server doesn't [01:39:47] hm [01:40:03] I had to restart nova network twice [01:40:14] it's a possibility that the NAT isn't working properly right noew [01:40:19] since it's reapplying the rules [01:40:26] It's also rejecting my public key from my laptop, but working fine from my server [01:40:41] very likely due to NAT rules [01:40:43] one minute [01:42:10] And in my sillyness I already answered the PuTTY prompt [01:42:15] So I can't tell you what it showed as the new host key :P [01:42:16] hm [01:42:18] it lets me in [01:42:24] ah [01:42:32] I think I know the issue [01:42:35] Ahh, now it's showing host key changed again [01:42:37] its key didn't change [01:42:43] Ryan_Lane: I had an account on the older Wikitech, and it seems to exist on the new, but what do I use for "Token" when logging in? [01:42:57] Jasper_Deng: If you didn't set up two-factor auth, nothing [01:43:08] your old wikitech password will not work [01:43:23] I imported content, not users [01:43:47] Ryan_Lane: Now it works [01:43:49] * FastLizard4 shrugs [01:44:03] yeah, it was due to nova network being restarted [01:44:04] hhm it said that my password was wrong, implying that my acccount was imported [01:44:18] when that occurs the NAT rules temporarily go away [01:44:30] Jasper_Deng: do you have a new wikitech account? [01:44:36] it turns out I don't [01:46:24] you can create a new one [01:46:39] does labs now have an automatic query killer? [01:46:40] the "incorrect password" error was misleading in this sense [01:46:54] because my screen session just had a "Terminated" [01:48:41] legoktm: on which instance was it running? [01:48:46] bots-bnr1 [01:49:08] i was doing a dump scan that might have used a lot of memory/cpu [01:49:19] but i had been doing them before for the past few days with no issues [01:49:58] as far as I know it doesn't [01:50:15] and I don't see an OOM in the dmesg [01:51:08] hmm ill ask petan then since thats a no root instance [01:51:36] Well, this day had shitty productivity. I'm calling it. [01:51:38] * Coren waves. [02:09:32] Ryan_Lane: oh i forgot, can you create a mysql user for me on bots-bsql or does petan have to do it? [02:09:45] petan I believe [03:06:58] !log account-creation-assistance Running apt-get upgrade on all server [03:06:58] Logged the message, Master [03:11:10] !log account-creation-assistance All servers requesting reboots, will provide [03:11:11] Logged the message, Master [03:51:03] !log bots addshore: legoktm user created on bsql01 [03:51:05] Logged the message, Master [03:51:09] :> [03:51:49] by the looks of things everyone with an account on bots already has a user on bsql01 [03:51:59] but I am not sure what it sets the passwords too O_o [04:15:31] @notify petan [04:15:31] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [04:15:34] o.o [05:33:04] addsleeep: [05:33:05] mysql> create database persondata; [05:33:05] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database 'persondata' [06:26:04] mysql> CREATE TABLE persondata ( page VARCHAR(255), qid INT, param VARCHAR(255), value VARCHAR(255)); [06:26:04] Query OK, 0 rows affected (14.75 sec) [06:26:13] Maybe its just me, or did that take too long? [09:03:10] boxofjuice hi [09:03:15] hi [09:03:18] what's up [09:03:26] well i got most things figured out [09:03:27] but [09:03:35] :o [09:03:38] is there a way i can limit how much memory a program can use? [09:03:42] I killed ur query yesterday [09:03:52] boxofjuice yes if you control source :P [09:03:59] just watch how much memory are you using :D [09:04:06] heh.... [09:04:12] i'm running a dump scan for someone [09:04:13] for example http://bots.wmflabs.org/~wm-bot/db/systemdata.htm [09:04:17] aha [09:04:24] okay... [09:04:29] across all revisions on all wmf projects [09:04:58] and apparently my script is rather greedy :/ [09:05:20] i'm not really sure how to control it [09:05:36] i just pulled it out of the pywikipedia repository (that should say enough) [09:05:37] ok man bash [09:05:40] ulimit [09:05:59] ulimit is a built in command in the shell. It will restrict any processes from exceeding the set criteria. You have the following options, memory included [09:06:11] core file size (blocks) 0 [09:06:12] data seg size (kbytes) unlimited [09:06:12] file size (blocks) unlimited [09:06:13] max locked memory (kbytes) unlimited [09:06:15] max memory size (kbytes) unlimited [09:06:16] open files 1024 [09:06:17] but is it going to kill my script when it goes over, or what? [09:06:18] pipe size (512 bytes) 8 [09:06:19] stack size (kbytes) 8192 [09:06:20] cpu time (seconds) unlimited [09:06:20] max user processes 1024 [09:06:22] virtual memory (kbytes) unlimited [09:06:22] yes [09:06:25] hmmm [09:06:34] i dont want to kill it, just a way to limit it. [09:06:41] im guessing that has to be done in the program itself. [09:06:49] okay in that case you need to do it from source code [09:06:51] yes [09:07:08] maybe that process has a parametere [09:07:13] ? [09:07:15] try --help [09:07:33] lol no its not that advanced [09:07:40] in that case no [09:07:48] ill look into how python does this stuff... [09:10:20] k [09:10:23] !tu [09:10:23] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [10:14:53] [bz] (NEW - created by: Peter Bena, priority: Unprioritized - major) [Bug 45768] console doesn't show proper errors - https://bugzilla.wikimedia.org/show_bug.cgi?id=45768 [10:15:01] Ryan_Lane ping [10:15:06] Add rule [10:15:06] ? [10:15:06] Jump to: navigation, search [10:15:07] Failed to add rule. -- wtf? [10:15:13] can you tell my what it means? [10:15:21] what what means? [10:15:29] ah [10:15:33] failed to add rule? [10:15:34] it would be so cool if console told you more than just "Error happened" [10:15:52] why it fails to add a rule? what is wrong? [10:16:10] are you trying to add a source group and a normal rule? [10:16:29] you need to tell me what you are doing so that I can figure out why it isn't working [10:16:34] I clicked add rule [10:16:44] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaSecurityGroup&showmsg=setfilter [10:16:45] right, with what values set? [10:16:59] port range 3306 - 3306 [10:17:01] protocol tcp [10:17:06] cidr 10.4.0.0/21 [10:17:11] source group default [10:17:29] "Note: Individual rules and group rules are mutually exclusive. Choose one or the other, not both." [10:18:06] so by filling this I don't assign the rule to specific group? [10:18:18] source groups don't do what you think they do [10:18:28] "Instances in added security groups will be allowed ingress of all ports and protocols." [10:19:16] if instance a and b are in security group web, and you have a rule that uses source group web, those two instances will be able to talk to each other on all port [10:19:17] *ports [10:19:50] we allow default, which allows all instances to talk to each other on all ports within a project [10:20:12] ah [10:20:17] normal security group rules define how things outside of a project can talk with instances in a project [10:20:25] anyway it would be nice to have error message that tell you details [10:20:28] it's honestly a confusing concept and it's awkward [10:20:40] in this case the passed through message is just as worthless as ours ;) [10:20:50] because I've been incredibly confused about this [10:21:31] really we should have error checking before it ever sends it to nova [10:21:52] if mutually exclusive rules are added together the interface should just deny and give a proper error message [10:50:55] !log wikidata-dev wikidata-dev-9: Moved devrepo's debug log from /srv/logs/devrepo.log to /var/log/wikidata-devrepo-debug.log. Added cronjob that deletes it once a day. [10:51:00] Logged the message, Master [11:13:35] Is there anyone with puppet merging super powers here right now? [11:21:00] paravoid for example? [11:41:17] ... 36% done... [12:16:20] petan: ping me if your around :) [12:56:12] addshore here [12:56:39] is there some sort of nice list with all of the parameters in for mariadb? :P [12:57:09] as far as I cant tell some system vars a different to regular mysql :/ [13:07:08] nice xD https://kb.askmonty.org/en/sample-mycnf-files/ "Place holder for sample my.cnf files" [13:28:30] * addshore found it [13:52:17] !log bots addshore: optimize bsql01 my.cnf to utilise more server resources [13:52:19] Logged the message, Master [13:52:45] use more server resources? [13:53:27] yes :D [13:53:52] its a big server but the config didnt reflect that [14:08:50] is bastion down? [14:15:51] JasonDC, appears to be [14:15:54] JasonDC, try bastion2 or bastion3 [14:16:01] the keys are broken again :/ [14:20:35] lol [14:20:55] bastion2 works [14:21:18] yep :) [14:21:22] !bastion [14:21:23] http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org which should resolve to 208.80.153.194; see !access [14:21:55] hehe PROBLEM SSH is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.0.54 output: Server answer: [14:21:57] !bastion del [14:21:57] Successfully removed bastion [14:22:41] !bastion is http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org (208.80.153.194) bastion2.wmflabs.org (208.80.153.202) see !access [14:22:41] Key was added [14:22:50] mhm [14:24:01] petan: and bastion3 ;p [14:24:36] no lol it's gluster this time [14:24:41] I am still on bastion1 [14:24:48] it's not possible to open /public/keys [14:25:14] Ryan_Lane ^^ [14:25:30] labs should have more admins :/ [14:25:40] so bad they only recruit paid employees [14:25:55] this is something what suck on wikimedia most [14:26:02] it is so wmf-centric [14:26:07] xD [14:26:24] * addshore is also on bastion1 still [14:26:30] seriously... it's not open source project anymore, it's owned by a company [14:27:21] petan: these are not mutually exclusive... [14:27:31] I know but it suck anyway [14:27:44] Linux is mostly improved by employees these days [14:27:46] it's not community based project [14:27:59] gribeco linux is completely developed by paid developers these days [14:28:02] that suck as well [14:28:23] it's almost impossible for regular volunteer to contribute to kernel [14:28:41] just as it's more and more impossible for volunteers to contribute to wikimedia project [14:28:59] that's why company based open source projects are bad [14:29:07] community is better and friendlier [14:29:13] and more open :> [14:30:40] :D [14:31:09] that's why I spend more time these days working on my own open source project rather than wikimedia :P [14:31:17] despite I am only member, it's still more open :D [14:31:41] petan, its great, as far as I can tell I increased caches and buffers a bit on bsql01 (one I actually increased 10 fold) and now it is using less ram O_o [14:31:43] project(s) [14:31:59] addshore did you restart sql server after that? [14:32:05] yep [14:32:17] addshore did you check if people were using it? :P [14:32:27] *didnt restart, reloaded* [14:32:28] because it's a very bad idea to kill server when it's being used [14:32:30] ah [14:32:38] I think you need to restart it for these to take effect [14:32:49] you can't change ram size while it's running [14:32:52] hmm, but after the reload I saw a decrease in mem usage :P [14:33:02] mhm... who knows [14:33:29] I bet mysql is using garbage collectors just as other software does which kind of suck [14:33:49] these are things that make it simple for devs but produced binaries mostly suck more [14:34:04] all my programs written in c# eat 200 times more memory than my programs written in c++ [14:34:23] like typical windows application in c++ eats around 2 mb of ram [14:34:30] typical c# 400 mb [14:35:51] right i shall be back later when WD rolls out ;p have a nice day! [14:36:02] addshore how's your query? [14:36:04] how you check it? [14:36:11] I don't see any progress bar :D [14:36:11] it was only at 30% so I killed it [14:36:22] I left it running in a screen [14:36:22] ok how you check it's in 30% [14:36:23] petan, define typical [14:36:32] and it shows me a % of how far through it is [14:36:43] let's say "simple" [14:36:43] almost tempted to go and run it again now I have altered my.cnf [14:36:56] eg application that doesn't need more resources than these to display its windows [14:37:08] like bunch of simple forms [14:37:16] as someone with 10 years of Windows development both in C and C#, I disagree [14:37:29] 10 years of development in c# [14:37:39] I am unsure if c# exist for so long? but maybe it does [14:37:50] anyway from my experience it eats more memory than c++ programs [14:37:57] not only windows application even terminal apps [14:38:02] * applications [14:38:18] if you write 2 programs doing exactly the same thing in both languages, the difference will be nowhere near that [14:38:51] of course, hello world in C# will eat prbably 30 megs due to runtime [14:39:22] but other than that, it's going to be a sane difference [14:39:29] MaxSem that's my point... two application doing the same will have very different need for resources [14:39:39] well, not 200 times more, ofc [14:39:46] but not 1:20; [14:39:51] 0 [14:40:37] but for example very simple tcp server I made in c# is eating about 100mb, the almost same application made in c++ eats about 10mb... I don't know why [14:40:47] wm-bot is eating 300mb+ [14:40:52] it's just a simple irc bot [14:40:55] of course, it's always possible to use a pathological behaviour in GC to achieve 400 megs working set size, but you'll have to try really hard [14:41:05] http://bots.wmflabs.org/~wm-bot/db/systemdata.htm [14:41:14] 306272kb [14:41:24] I can't understand what needs to much memory in that bot [14:41:37] very little number of strings and some classes [14:41:44] basically everything is flushed to disk [14:42:07] dotTrace? [14:42:11] even if it wasn't - all datafiles that are loaded have 3.7mb on disk [14:42:22] even if I loaded them all to memory it should eat 3.7mb of ram [14:42:39] MaxSem no idea what is dottrance [14:42:41] * trace [14:42:52] but I really do miss "delete" [14:42:53] good typo [14:42:55] in c# [14:43:14] it's a profiler [14:43:29] ok I have had one profiler verfy good - ants [14:43:36] I used it on one of my application and discovered nothing [14:43:46] 80% of allocated memory was used by .net internals [14:43:50] I used it mostly for perf stuff, but it also has memory profiling [14:45:07] I don't even think it's technically possible for GC to handle memory better than programmer - it can either handle it same good as programmer, or better in case programmer suck [14:45:22] it's not possible [14:45:30] what is not possible [14:45:52] for GC to manage memory better than human [14:45:55] HOWEVER [14:46:07] to err is human [14:46:16] More to the point, it's not possible for a FC to handle memory better than a human /can/. [14:46:20] GC* [14:46:20] in that case I don't see many advantages in that, other than that it's less error prone [14:46:22] thus, GC is there to prevent memory leakks [14:46:38] yes I know [14:46:59] but it should be possible to opt-out from using it, otherwise... I am afraid c# had a very bad future [14:47:06] * has [14:47:30] perfect code in c++ will be always faster and less memory expensive than code using GC [14:47:41] even if that code was perfect as well [14:47:42] ever wrote a multithreaded program with boost::shared_ptr?:P [14:47:48] nope :P [14:48:01] oooh, what a lucky man [14:48:02] I have made a lot of c++ programs with memory leaks though [14:48:37] but still I believe c# should have possibility to disable GC and let human control memory [14:48:45] MaxSem: If you inflict boost on yourself, you deserve the result. :-) [14:49:14] Coren, the result was me moving to WMF:0 [14:49:31] YES I DESERVE IT [14:49:35] :-) [14:50:01] if used sanely, boost isn't that bad [14:50:07] I like C++ very much, but I use it with the "C with classes" idiom. That generally means staying the fuck away from std. :-) [14:50:22] boost is right out. [14:50:36] and C++0x actually makes a lot of STL/Boost code sane [14:51:23] Meh. People overuse the OO idiom anyways. It only maps to a very limited problem space in practice. [14:51:37] Classes as code encapsulation / interface definition is Good. [14:51:53] C++'s idea of classes is bad [14:52:00] I know:) [14:52:03] MaxSem: iff you try to use it for OO [14:52:34] As syntactic / organization sugar over C, they're very good. [14:53:38] as soon as you need a container class, you either have to store pointers and manage memory yourself, or use that automagical machine gun for your feet called assignment constructor [14:54:25] MaxSem: True; which is why I have my own container idiom handy for reuse when I need it. [14:54:51] using the NIH pattern?:P [14:55:01] am i the only one who cannot login to the bastion right now? [14:55:05] no [14:55:47] Heh. No, using the "there's no reason to have a complicated solution when a simple one will do" pattern. :-) [14:56:09] Silke_WMDE_: Hm. Broken for me too. I shouldn't have closed that ssh I already have opened. [14:56:25] Coren, do you already have powers to fix it? [14:56:37] MaxSem: Nope. [14:56:54] * MaxSem bites the ops team [14:56:57] Coren: So did I when switching to wifi a few minutes ago. m) [15:00:08] MaxSem: I have two generic classes I use for my code; a pointer-to-something and a reference-counted-pointer-to-something, both with threadsafe variants, an a discardable allocation pool class. Between those five, I solved 99.9% of all my needs. [15:00:33] sigh [15:00:39] classic C++ [15:00:49] Why the sigh? [15:01:14] there's a standard library, but it does so little and with so many restrictions that nobody uses it [15:02:11] MaxSem: That's an unavoidable consequence of most people trying to use C++ for the wrong thing, not of the language design. If you need Java, use Java. :-) [15:03:14] and if I need C's performance AND C#/Java's qability to handle huge piles of code? [15:03:50] that was a typical task at my previous work [15:04:20] several epic architects failed at it, some were fired [15:05:18] You need to sacrficice something then. I'd also like a computer with ten times the performance of my current desktop and I have a budget of $100. :-) [15:05:45] FAIL [15:06:53] Coren: I call that powerful computer "WMF labs" ;) [15:07:27] (And I'd dispute the putative quality of C#/Java to "handle huge piles of code" any better than any other tool. Coder discipline cannot be compensated for by the language.) [15:07:43] boxofjuice, and gluster be upon us!:P [15:07:46] boxofjuice: I'm fairly confident the WMF had a budget slightly bigger than mine. :-) [15:09:13] Coren, the problem is that in C/C++ you tend to struggle a lot with language's features while the more scriptish a language is, the more you can dedicate yourself to actually doing the shit done. but everything come with its own price [15:09:29] hmmm [15:09:47] is bastion broken? [15:09:54] aude: yup :/ [15:09:58] MaxSem: Of course. This is why perl remains a weapon of choice in my arsenal. [15:09:58] * aude reads above... [15:09:59] ok [15:10:17] eww, I hate perl [15:10:25] MaxSem / Coren: since you guys were discussing memory earlier, would you like to help me figure out how to make a python dump parse script i'm using not use as much memory as possible? [15:10:52] boxofjuice: My python-fu is weak. [15:10:56] my python is very noobish, unfortunately [15:11:04] :( [15:12:05] MaxSem: Perl is really good at what it does. For the typical use case, I'm usually done with implementation before someone else finished loading their IDE. :-) [15:12:35] it's too schizophrenic for me [15:13:49] MaxSem: The biggest hurdle is that it's /highly/ idiomatic. [15:14:26] @#Y$@#$Y#@$Y@#$^#$^#@$^@# [15:14:41] a typical perl one-liner [15:15:09] MaxSem: Also a very /mean/ one. IOCCC has nothing on perl. [15:15:11] sorry, should've included a couple of ~s [15:15:22] MaxSem: But it /is/ possible to write very readable and maintainable perl. [15:16:15] MaxSem: The biggest difficulty stems from the fact that reading perl requires the same level of still writing perl did; something that is generally not true with other languages. [15:18:13] while(<>) { stuff if m/crap/; } is highly expressive and understandable once you know the idioms. [15:20:18] And you gotta love a language where you can say "open somefile or die;" :-) [15:22:35] !log bots petrb: updated buffer size of sql server, was too big [15:22:38] Logged the message, Master [15:51:08] Coren: MaxSem fwiw, the Socialtext wiki is in Perl, and it is (or at least was) one of the better code bases I've worked in. [15:52:03] some people even write production code in Erlang, so what? [15:53:11] MaxSem: It's just illustrative that code quality doesn't depend on the language but on the coders. :-) [15:56:02] Ryan_Lane: Gimme the bits so I can fix bastion when it is broken! [16:10:06] Coren: Will you go to the Hackathon? [16:10:29] Darkdadaah: Ayup [16:10:55] Ah good, I'll probably go this time :) [16:13:33] labs down? [16:13:38] *bastion down? [16:14:32] ssh_exchange_identification: Connection closed by remote host [16:14:39] bastion1 is [16:15:10] JasonDC: so which one can I use? [16:15:11] oh, and keys are having issues: Permission denied (publickey). [16:15:16] bastion2 or 3 [16:19:15] JasonDC: so you were describing the issue when ssh'ing from bastion2/3 to instances? [16:19:24] ya [16:21:36] ok I'll just wait... [16:24:36] Coren, you can say the same thing in php... [16:25:37] Platonides: So you can. [16:37:13] Coren: Will it be long/difficult to add user databases to Tools? [16:38:29] Darkdadaah: Probably not, though round one will probably be a transitory method. [16:38:37] Darkdadaah: (Mysql on a project instance) [16:40:06] Did you settle for a mean of identification? [16:41:21] Darkdadaah: That's what we'll be doing today, time allowing. [16:45:28] I'll just wait until it's ready then. Thanks for your work. [16:56:31] Damianz I sorted out the self creation of db's but I need you to insert one line to your script [16:56:48] grant select, execute on system.* to user; [16:56:58] where user is a new user you are making [16:57:29]