[01:38:43] legoktm: I think the problem with afch-updater's failing grid job submissions is that you use /usr/*local*/bin/jsub, and on webserver-01, that is an obsolete version. I'll fix that, and we need to put it in Puppet probably for those people who rely that it is located in that directory. [01:51:34] [bz] (8NEW - created by: 2Tim Landscheidt, priority: 4Unprioritized - 6normal) [Bug 52258] Obsolete scripts in /usr/local/bin need to be "managed" - https://bugzilla.wikimedia.org/show_bug.cgi?id=52258 [01:51:50] Ryan_Lane: It lives. Its annoying stalls have gotten no better nor any worse. I wanted to switchover to labstore4 to eleminate hardware failiure as a possibility (at least the controller's) but I'm not keen on making a change of that magnitude just before Wikimania. [02:01:48] !log tools tools-webserver-01: Symlinked /usr/local/bin/{job,jstart,jstop,jsub} to /usr/bin; were obsolete versions. [02:01:52] Logged the message, Master [02:24:56] !log wikistats added missing el.wikivoyage, fixed wikivoyage timestamps/updates, running wiki-site update, add 1.22wmf12 to good versions [02:24:58] Logged the message, Master [02:28:39] Coren: understandable [04:25:45] !demon [04:25:46] <^demon> Docs exist solely for developers to go "omg you didn't read the docs!" when people ask common questions. In practice, nobody reads docs before asking questions. [06:22:59] petan? [07:16:16] Petan?? [07:16:32] addshore: [07:16:48] Or coren :) [07:17:42] Hi yuvipanda [07:17:56] * YuviPanda waves [07:21:21] hi [07:21:27] @notify addshore [07:21:27] This user is now online in #huggle. I'll let you know when they show some activity (talk, etc.) [07:30:22] Pet an [07:30:32] *petan [07:30:33] add hore [07:30:44] heya [07:31:03] Can you "qdel -u 'local-addbot-" for me ? :) [07:31:26] you can't? :o [07:31:29] Without the extra dash ;p [07:31:42] I'm on my phone! :( [07:32:00] !log tools petrb: deleted local-addbot jobs [07:32:03] Logged the message, Master [07:32:33] Cheers :) [07:33:08] *stop walking towards the internet like a mad man* [07:33:15] *stops [07:49:33] &info [07:49:34] http://tools.wmflabs.org/wm-bot/dump/%23wikimedia-labs.htm [08:23:56] hashar: is there installation of ZMQ on beta? [08:24:03] or something we can test that new code on [08:24:16] lo [08:24:23] lo? [08:24:30] hello [08:24:32] :D [08:24:35] aha :D [08:24:39] petan: I asked for you first [08:24:44] there is an event logging instance in beta [08:24:50] zhuyifei1999 did you? [08:24:53] zhuyifei1999: what's up [08:25:06] (06:22:59) petan? [08:25:08] so there must be some zero mqueue install on it. MaxSem or Ori-l would know about the EventLogging in beta -:] [08:25:12] hashar: how does it work is it documented? [08:25:26] zhuyifei1999 ok that isn't really a question [08:25:37] hopefully it is setup the same as in production so the prod documentation should apply to beta as well [08:25:41] you need to add a text between petan and ? [08:25:46] petan: Why aren't you in offtopic? [08:25:48] hashar: ok [08:26:01] idk [08:29:00] petan: and a bug: https://tools.wmflabs.org/?Rules of the sidebar of https://tools.wmflabs.org/ links to a non-existing page [08:31:22] zhuyifei1999 I know, [08:32:13] fixed XD [08:37:39] [bz] (8NEW - created by: 2Amir E. Aharoni, priority: 4Unprioritized - 6normal) [Bug 52249] localization messages loading issues in http://en.wikipedia.beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=52249 [10:09:23] petan: do you know how to diyplay, how mandy log-actions (filemoves etc.) a user has? [10:09:33] Hm... is this possible via api? [10:23:48] Steinsplitter: I think so [10:24:31] petan: What's the rules fot Tool Labs? [10:24:37] ^^ [10:24:49] Steinsplitter: ? [10:24:58] hwo to? [10:25:00] *how [10:25:04] *of [10:25:29] Steinsplitter: finding... [10:25:41] like this https://github.com/Pathoschild/Wikimedia-contrib.toolserver/tree/master/stalktoy O_O [10:25:48] but work with db :P [10:27:24] :/ [10:27:39] so much for an interactive session :P [10:27:48] where did tools-login go? :P [10:28:27] I think nfs is dead [10:28:28] &ping [10:28:29] Pinging all local filesystems, hold on [10:28:30] Written and deleted 4 bytes on /tmp in 00:00:00.0002330 [10:28:33] see [10:28:34] :/ [10:28:42] petan: can i sql my jobs? [10:28:45] *qdel [10:28:53] I don't think so [10:28:56] :< [10:29:02] Written and deleted 4 bytes on /data/project in 00:00:32.6104720 [10:29:09] 32 seconds for 4 byes [10:29:10] bytes [10:29:15] &ping [10:29:15] Pinging all local filesystems, hold on [10:29:16] Written and deleted 4 bytes on /tmp in 00:00:00.0005060 [10:29:17] Written and deleted 4 bytes on /data/project in 00:00:00.0054240 [10:29:18] managed it :) [10:29:25] everything just came back to life xD [10:32:52] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.0.16 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [10:34:11] Reliability of the tools cluster seems to be suffering these days; too much unmanaged change, or new problems manifesting as the system is excercised by new services and users? [10:34:48] Or are we still in an alpha enviroment really? [10:36:11] heh, it always seems to recover well [10:36:57] im just interested to know what has been happening on the webserver xd http://ganglia.wmflabs.org/latest/?c=tools&h=tools-webserver-01&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [10:40:01] petan: no doc about &ping [10:54:50] &ping [10:54:51] Pinging all local filesystems, hold on [10:54:52] Written and deleted 4 bytes on /tmp in 00:00:00.0006030 [10:56:11] Written and deleted 4 bytes on /data/project in 00:01:20.1649200 [10:56:26] &ping [10:56:26] Pinging all local filesystems, hold on [10:56:27] Written and deleted 4 bytes on /tmp in 00:00:00.0002040 [10:56:28] Written and deleted 4 bytes on /data/project in 00:00:00.0057190 [11:07:19] petan: I am no officily off the old db :> [11:07:38] english please :P [11:07:55] i dont need bots-bsql01 ;p [11:08:00] anyway, there is a bunch of other people who do [11:08:11] I need to get them off [11:08:11] :> [11:08:16] Damianz: you are one of them :P [11:08:30] also, is there a limited number of jobs on the grid per tool? [11:08:33] petan: -.- [11:08:36] yes [11:08:38] i seem to remember ebing told before :P [11:08:39] Damianz: no worries [11:08:40] what is it [11:08:41] ? [11:08:48] Damianz: in fact I don't care, it's Ryan who want to have it deleted [11:08:58] addshore: like 15 [11:09:08] that means 15 running, you can have a bunch of waiting [11:09:09] yup, 16 makes sense :/ [11:09:10] I'll see if I have time this weekend [11:09:13] or 16 [11:09:17] Damianz: no problem [11:09:22] shame I cant make this go any faster then :< [11:09:31] Damianz: I have vacation next 2 weeks [11:09:35] I should write a mail [11:09:53] I have like omg work to do for the next 2 weeks heh [11:13:57] addshore: hehe, too much power? :P [11:14:16] mhhm, id like to get this run over the db done as fast as possible :P [11:14:25] but I can only have 15 jobs ;_; [11:14:47] ;_; indeed :P [11:14:50] im sure there is a way to reduce the number of jobs to one :P [11:14:51] only 15 jobs :P [11:15:09] addshore: what exactly are you doing? [11:15:22] addshore: also is this now 15 times faster than doing it in one job? [11:15:54] yus :> [11:16:04] :D [11:16:05] but i tihnk there is a way to group things on oge [11:16:30] YuviPanda: I have 1.4 million ish thing to check, and it takes me 6 hours to do about 50k ish [11:16:46] addshore: ah, hmm. How much of that is IO vs computation? [11:16:53] addshore: are you doing lots of database work? [11:16:58] or network work? [11:17:14] probably half network and half computational [11:17:18] and 1% db work [11:17:33] hmm [11:17:59] as the script runs for longer the number of network requests decreases though ;p [11:18:18] addshore: 6 hours to do 50kish with the 15 jobs? [11:18:29] yup, i left it running last night :) [11:18:39] ah :) [11:18:42] thats a very rough estimate ;p [11:18:47] addshore: perhaps ask Coren to lift the limit? :P [11:19:02] alternatives being multi threading (co-operative or otherwise) [11:19:08] but that's going to be very, very painful in PHP :P [11:19:09] [bz] (8NEW - created by: 2Amir E. Aharoni, priority: 4Unprioritized - 6normal) [Bug 52222] fill http://he.wikipedia.beta.wmflabs.org/ with some useful data from he.wikipedia.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=52222 [11:19:16] hmm, actually more than 50,000 [11:19:44] more like 150,000! [11:19:52] hmm [11:19:58] that's 600,000 per day [11:20:03] thats fine actually, ill be done in 2 days ;p [11:20:08] so that's still just 2 days [11:20:09] yeah ;) [11:20:24] addshore: you brought it down from 30 days to 2 days with Redis! :D [11:20:30] :> [11:20:48] addshore: write up about this to labs-l, no? other people can probably also use similar techniques [11:23:06] Coren: about the nfs problems: a friend of mine thinks that openafs or ceph might be suitable nfs replacements [11:23:19] http://www.openafs.org/ http://ceph.com/ [11:23:27] JohannesK_WMDE: we've Ceph in production for... something. Not too favorable opinions of it, IIRC [11:23:38] (we used Swift before, and not too favorable opinions of that either) [11:23:38] i haven't tried them myself, but might be worth a look [11:26:43] something must be done, i get horribly annoying delays all the time. maybe try openafs if ceph isn't suited [11:27:14] logCursor.execute('INSERT INTO logs VALUES (?, ?)', (timestamp, unicode(str(*args).decode('utf-8')))) [11:27:14] OperationalError: disk I/O error [11:27:29] ^^ speak of the devil [11:27:33] Coren ^^ [11:27:47] &ping [11:27:48] Pinging all local filesystems, hold on [11:27:48] Written and deleted 4 bytes on /tmp in 00:00:00.0005930 [11:27:50] Written and deleted 4 bytes on /data/project in 00:00:00.0081680 [11:27:58] I think Coren was about to replace the controller this thing was on [11:28:49] what kind of controller? [11:28:52] disk [11:28:55] ok [11:36:55] hah petan There are few trained monkeys who know how it works. But unfortunately [11:36:55] not many of them, [11:37:10] Remember: if nothing works, typically killing the bot entirely works :-) even if it's evil. [11:41:55] JohannesK_WMDE: You're missing the point of the issue; NFS stalling is the /symptom/ the the controller becoming unresponsive until it gets soft reset by the kernel. NFS - EXT4 - LVS - LVS - RAID - RAID - [ driver - hardware ] <- only the last has issues. [11:42:32] i didn't know that. *you* talked about NFS having issues. [11:42:48] if it's the hardware, it should be repaired or replaced... [12:38:19] JohannesK_WMDE: you have an idea how hard is it to repait a hard disk :-) like you need a McGiver to fix it with his screwdriver or something [12:39:15] %ping [12:39:19] wait xD [12:39:21] &ping [12:39:21] Pinging all local filesystems, hold on [12:39:22] Written and deleted 4 bytes on /tmp in 00:00:00.0002900 [12:40:05] Written and deleted 4 bytes on /data/project in 00:00:44.1246410 [12:44:09] petan: sonic screwdriver or magnetized needle usually works best [12:50:36] YuviPanda: what's redis function to rbpull [12:50:39] or whatever it was [12:50:40] in python [12:50:59] rbpop? [12:51:36] redis has no attribute rbpop [12:55:37] [02dispatcher-labs] 07benapetr pushed 031 commit to 03master [+0/-0/±3] 13http://git.io/jKqMYw [12:55:39] [02dispatcher-labs] 07benapetr 0336938c2 - implemented some more verbose output [13:01:11] petan: what's &ping? [13:01:33] [02dispatcher-labs] 07benapetr pushed 031 commit to 03master [+0/-0/±1] 13http://git.io/N1lCPQ [13:01:34] [02dispatcher-labs] 07benapetr 03407a2ef - fixed some issues with save / load [13:01:43] &help [13:01:43] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.0.16 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [13:01:53] @ping [13:02:07] &help ping [13:02:08] Unknown command type @commands for a list of all commands I know [13:02:48] petan: http://redis.io/documentation [13:02:58] petan: to be more exact, http://redis.io/commands [13:03:08] you'll see that it is called 'brpop' [13:03:20] YuviPanda: I meant python command name [13:03:26] should be the same. [13:03:33] k [13:03:43] most clients mirror the official redis command names [13:07:48] [bz] (8NEW - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 52275] Status page should automatically refresh data - https://bugzilla.wikimedia.org/show_bug.cgi?id=52275 [13:08:51] petan: what's &ping? [13:08:59] it pings the disk [13:11:30] <{{Guy}}> :) [13:11:36] <{{Guy}}> !ping [13:11:36] pong [13:16:29] !ping [13:16:29] !pong [13:18:30] [02dispatcher-labs] 07benapetr pushed 031 commit to 03master [+0/-0/±4] 13http://git.io/XOSL4A [13:18:31] [02dispatcher-labs] 07benapetr 0325d8f0d - fixed missing check in Stream reader and some more [13:35:20] !ping [13:35:20] !pong [13:35:45] whoms bot is Not-002 ? [13:36:49] YuviPanda: you know? [13:37:03] AzaToth: petan [13:37:14] <{{Guy}}> It's the github bot, no? [13:37:17] could figure ツ [13:37:42] * AzaToth hates people who changes their nick [13:38:05] {{Guy}}: yes it is [13:38:11] unregistered offcoure [13:38:37] <{{Guy}}> What's unregistered? Not-002 ? [13:38:42] yup [13:39:09] and I can't parse the name [13:39:13] Not-002? [13:39:17] what does it mean? [13:39:18] petan: ? [13:39:42] notification [13:39:58] yes? [13:40:05] ah [13:40:14] can you please speak like petan: ? [13:40:20] not just petan? [13:40:25] petan: ↑? [13:41:08] <{{Guy}}> petan butter and jelly? [13:41:53] petan: I couldn't understand what Not-002 meant [13:42:03] not double zero two [13:42:03] it's a giyhub bot [13:42:06] * github [13:42:13] second notification bot [13:42:19] it is just like grrrit-wm, but it works for github repositories [13:42:28] petan: you might want to reg it [13:42:35] I can't it's not mine bot [13:42:36] second gerrit notification bot [13:42:43] it's a github service [13:42:47] k [13:43:00] anyway, I don't use gerrit because it takes months to get a repository set up [13:43:07] I need some system that is just as easy as github [13:43:30] petan: usually it takes a couple of hours max [13:43:37] you just need to poke the right person ツ [13:43:49] last repository I requested I had to wait more than 3 months for [13:43:53] lol [13:44:03] I was poking Chad frequently [13:44:05] ^demon|zzz: you hear that? [13:44:18] but that was when git was starting... [13:44:22] I see [13:44:25] there was quite a backlog back then [13:44:34] I assume that could have been a issue [13:44:50] anyway I just didn't like the mere fact that I have to ask someone else to create a repository instead of making it myself like on github [13:45:17] petan: well, it's prettyu logical from a basic point of view [13:45:58] can't just allow anyone to create a repo at any time with any name containing anything [13:47:45] as it's wmf's they need to have some control [13:50:37] YuviPanda: perhaps we should have colorized [repo/bar] as well [13:52:59] * YuviPanda reads scrollback [13:55:56] YuviPanda: would be nice to have the repo name as the first item of grrit bot notifications [13:56:23] (CR) JohnDoe: [C: 1] Some commit summary [repository/IdontcareAbout] .. [13:56:41] I got to read all the first bits before figuring out the change happened in some random extension :] [13:57:05] hmm, we could put that right after (CR) [13:57:13] AzaToth: thoughts? [13:57:23] we could even put it in the bracket [13:57:29] (CR: mediawiki/core) [13:57:44] addshore: did you notice your 2 processes on -login were autokilled [13:57:55] the e-mail is wrong [13:57:56] what processes? [13:58:11] you had 2x php process each of them using 1.6g of ram [13:58:28] they were automatically killed but the e-mail is wrong :P [13:58:34] ohhhh!!! yes :> [13:58:37] it's the other way [13:58:46] I was 'activly debuggin' and managed to debug it into a never ending looop ;p [13:58:47] process ws 1.6g and free memory was 40mb [13:58:56] :> [13:59:04] it reported it as your process was eating 40mb and free memory was 1.6g [13:59:09] I need to fix that template [13:59:23] where does ti report it? :O [13:59:34] to your e-mail [13:59:36] type mutt [13:59:45] it does? :P [13:59:46] unless you forward it [13:59:48] which email? :P [13:59:56] e-mail of owner of process [13:59:58] so your tool [14:00:07] heh, never checked my tool email ;p [14:03:22] YuviPanda: yea, that could be fine [14:32:08] yoo andrewbogott [14:32:10] you there? [14:34:03] ottomata, what's up? [14:34:05] I think there's a problem with role::puppet::self since the puppet repo refactor stuff [14:34:17] getting Could not find class passwords::puppet::database for i-00000861.pmtpa.wmflabs at /etc/puppet/manifests/base.pp:84 [14:34:43] yeah… there's kind of a chicken/egg problem, I'm not sure how to automate it. [14:35:05] If you add this line to your puppet.conf it should work: modulepath = /etc/puppet/private/modules:/etc/puppet/modules [14:35:22] Oh, and you'll need to update your private repo as well. [14:35:24] hmmmmm [14:35:45] Yeah, not ideal. [14:35:55] Actually, just updating private repo might work, I haven't tried that [14:36:28] hmm, not sure if I have permissions to pull [14:36:36] shoudl I change rem otes? [14:36:41] origin ssh://labs-puppet@gerrit.wikimedia.org:29418/labs/private.git (fetch) [14:37:00] $ sudo GIT_SSH=/var/lib/git/ssh git pull --rebase [14:38:02] hm, ok, well, the puppet self stuff gets a custom puppet.conf [14:38:09] should it add that line to it when it gets setup? [14:40:03] oh, hm [14:40:09] andrewbogott: that line already exists in my puppet.conf [14:40:18] great! [14:40:26] Then updating private should do it. [14:40:28] ORRRR [14:40:29] hmm [14:40:30] wait no [14:40:31] hm [14:43:28] ahhh i see, shoudl work for new self hosted puppetmasters [14:43:33] just not ones that have already been deployed [14:43:33] got [14:43:34] it [14:44:14] well… I think it's reasonable to require that both repos be updated. It's not too crazy to have them make changes that depend on each other. [14:44:36] right sure, i was just confused [14:44:51] because I saw the modulepath line in my local puppet repo's 10-self puppet.conf [14:44:57] but not on my labs puppetmaster [14:45:08] but then I realized thats' cause you recently committed it [14:45:15] great, works now [14:45:16] oh… so maybe you do have to add that line by hand then. that's not great :( [14:45:20] yeah [14:45:28] it'll work for new setups [14:45:30] but not for old ones [14:45:43] because the new setups can't apply the change to puppet.conf unless they can run puppet [14:45:46] and they need that line to run puppet [14:45:50] sorry [14:45:52] *old setups* [14:47:34] hmmm, i dunno, something is weird, [14:47:43] it looks like one of my labs instances was just puppetized as if it was in production [14:47:48] got ganglia prod adddress, etc. [14:48:01] oo, ja that's not good [14:52:54] Coren: thoughts on zeroconf / avahi for toollabs? [14:52:54] or some form of service registry [14:52:54] to let tools communicate with each other over the network [14:53:13] YuviPanda: Not something I have any thoughts to at this time. Do you have a suggestion to make? [14:53:22] s/have/gave*/ [14:54:23] Coren: I'm looking at avahi, but am looking for an early -2 from you. Don't want to waste time on something that won't be implemented. [14:55:03] Coren: use case is to have continuous jobs that can connect to other ones without having to depend on dirty tricks like putting host and port number on a static file somewhere [14:55:34] Avahi is a reasonable approach to this, I wonder how well the OpenStack "networking" layer copes with multicast though. [14:55:57] Or, for that matter, /whether/ it does. [14:56:08] I don't think we need to have multicast for this. [14:56:14] we just need DNS-SD (service discovery) [14:56:18] not dynamic DNS [14:56:29] we perhaps should even explicitly disable dynamic DNS [14:56:41] anyway, considering that you think this is a reasonable approach, let me read up more on avahi :) [15:01:16] Coren: don't think we want to use Avahi. Has a DBus dependency [15:01:36] * YuviPanda looks for other ways to do service discovery [15:01:59] I don't mind dbus; it's a reasonable local delivery mechanism for events, and even supports remote bus connections. [15:02:55] yeah, but it makes me a little queasy. [15:03:07] plus we only need DNS-SD, not all of avahi. [15:06:56] man that feels too desktop centered :( [15:10:38] Coren: http://linux.die.net/man/8/sge_st [15:11:03] very poorly documented tho [15:11:51] That also doesn't seem to quite fit. [15:12:17] yeah [15:12:20] and seems rather abandoned [15:12:29] Honestly, I don't mind the desktop slant of either avahi nor dbus, and I can see a number of valuable uses for dbus anyways enough that I already considered it. [15:12:43] such as? [15:13:07] avahi requires the 'servers' to register manually, and the libraries for that seem unsupported [15:13:19] and very desktop oriented. [15:13:27] that, or the docs suck :P [15:13:50] I don't think you can get away from the registration requirement; I know of no discovery system that magically knows what a new daemon does. :-) [15:14:19] of course, but the wrappers that let you do registration should be simple enough. and maintained. [15:14:28] http://stackoverflow.com/questions/3430245/how-to-develop-an-avahi-client-server doesn't look like either :( [15:17:16] http://avahi.org/wiki/PythonPublishExample doesn't look so bad. [15:18:09] noooooo! :P [15:18:27] I'll probably just create files with appropriate perms for now. [15:20:54] i haven't used my labs account in over a year I think ; and i'm looking to try (again) to create an instance of wikistream to run in the labs environment under this project https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream [15:21:24] is there a good place for me to start to see what I need to do to set up and test an instance of the software? [15:21:48] at the moment it's running on my own personal server http://wikistream.inkdroid.org/ [15:22:07] and i'd like to transition it to run in the labs environment [15:22:33] also, hi :-) [15:22:35] * YuviPanda waves at edsu [15:23:15] edsu: https://wikitech.wikimedia.org/wiki/Help:Contents? [15:23:54] YuviPanda: thanks! i was somehow missing that :) [15:24:00] :) [15:34:35] well that was easy ... [15:34:42] I think I need some handholding. I have a patch to a MW extension, and I want to test it on a labs instance. I'm not sure how to start. [15:42:38] ragesoss: Vagrant is most likely what you need, then. [15:43:07]