[00:00:28] Oh, suck. libredis-perl is a year and a half old. [00:01:00] same for python-redis, btw. I always use virtualenv [00:01:17] I should talk to the powers that be at Wikimania and get rid of all system packages. pip all the way! [00:01:33] Not yet. :-) [00:01:41] not yet travelling? [00:01:52] my flight's in 22 hours, I've managed to completly fuck up my sleep cycle [00:02:19] I was about to start the packing and all, but then I realized that I've got some time to idle now. :-) [00:02:22] http://ganglia.wikimedia.org/latest/graph.php?r=2hr&z=xlarge&h=labstore3.pmtpa.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Labs+NFS+cluster+pmtpa [00:02:29] * yuvipanda clicks [00:02:35] So, writing less often makes the pain less. [00:02:36] nice buildings! :) [00:04:17] my python seems to have gone to shit :( [00:04:19] * yuvipanda refactors [00:05:44] (03CR) 10Yuvipanda: [C: 032 V: 032] Add support for registering for GitHub rsubscriptions too [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77559 (owner: 10Yuvipanda) [04:35:31] [bz] (8RESOLVED - created by: 2Peter Bena, priority: 4Normal - 6normal) [Bug 51936] Set up a server for query relaying - https://bugzilla.wikimedia.org/show_bug.cgi?id=51936 [04:38:51] [bz] (8NEW - created by: 2Johannes Kroll (WMDE), priority: 4Highest - 6critical) [Bug 52500] Broken Disk Controller is Broken - https://bugzilla.wikimedia.org/show_bug.cgi?id=52500 [06:38:05] petan: what's with this $6 stuff? [04:54:19 PM] Change on mediawiki a page Manual:Pywikipediabot/Gerrit was DrTrigon, changed by https://www.mediawiki.org/w/index.php?diff=756289 link /* Example (step-by-step) */ rcs-keywords â rcs-keywords.py edit summary: $6 [07:34:16] There seems to be a consistent load on the web server now. It's making my page, which is less that a kilobyte, take forever to load. [07:35:46] Hello? [07:37:16] CP678|iPhone: on the beka cluster? [07:37:23] *beta [07:37:27] Labs [07:37:34] Era tools [07:37:43] *Err [07:38:20] https://tools.wmflabs.org/ verry slow. yes. [07:40:15] Just look at the graph showing the load for tools. http://ganglia.wmflabs.org/latest/?c=tools&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [07:41:52] CP678|iPhone: maby opening a ticket [07:42:10] tools-webproxy [07:42:10] load_one: down [07:42:31] maby? [07:43:59] https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs [09:55:18] (03PS1) 10Krinkle: template: Remove underline from emphasis (only colour and bold) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77566 (via SuchABot) [10:04:14] (03CR) 10Yuvipanda: [C: 031] "Happy to Merge if AzaToth is okay with it, since he did the original Underline stuff." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77566 (owner: 10SuchABot) [10:12:54] hey AzaToth [10:13:05] i added you as a reviewer on one of Krinkle|detached's patches [10:13:30] (03CR) 10AzaToth: [C: 031] template: Remove underline from emphasis (only colour and bold) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77566 (owner: 10SuchABot) [10:15:01] hmm [10:15:32] YuviPanda|sleep: why is it on github? [10:15:41] why not? [10:15:42] :D [10:15:43] github is nice [10:16:03] (03CR) 10Yuvipanda: [C: 032] template: Remove underline from emphasis (only colour and bold) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/77566 (owner: 10SuchABot) [10:16:56] no automerge? [10:17:32] should [10:17:35] jenkins is just slow [10:17:56] or do I need to hit 'submit'? [10:18:08] hmpf, fine [10:18:13] heh [10:18:16] yeah, merged [10:38:12] AzaToth: tools.wmflabs.org is down [10:42:09] hey nerus [10:43:48] Hey somnambulistic YuviPanda ;) [10:43:53] :) [10:44:02] oh you are not asleep anymore. [10:44:05] no :P [10:44:08] stupid irc client [10:45:31] hehe :D [10:52:30] hmm [10:53:09] Steinsplitter: did you make a bug for it? [10:53:35] no. i [10:53:39] * Steinsplitter go to open a bug [10:55:11] &ping [10:55:11] Pinging all local filesystems, hold on [10:55:12] Written and deleted 4 bytes on /tmp in 00:00:00.0005980 [10:55:13] Written and deleted 4 bytes on /data/project in 00:00:00.0072820 [10:55:24] [bz] (8NEW - created by: 2Steinsplitter, priority: 4Unprioritized - 6blocker) [Bug 52523] tools-webproxy down - https://bugzilla.wikimedia.org/show_bug.cgi?id=52523 [10:58:06] addshore: http://tools-webserver-01.proxy.wmflabs.org/ works :P [10:58:15] haha! [10:58:26] addshore: http://tools-webserver-02.proxy.wmflabs.org/ too [10:58:42] yee, its just the proxy thats down [10:58:49] well, the other proxy... [10:58:55] yeah :P [10:58:59] apache down down! :P [11:00:07] Ryan_Lane, don't suppose you are around for any reason? :/ [11:00:15] doubt it :P [11:00:50] meh, worth a try [11:00:50] :> [11:00:52] :) [13:08:56] (03PS1) 10Yuvipanda: Hardcode path to the local virtualenv on toollabs [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77572 [13:14:03] (03CR) 10Yuvipanda: [C: 032 V: 032] Hardcode path to the local virtualenv on toollabs [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/77572 (owner: 10Yuvipanda) [13:42:08] Coren|Travel: Any reason as to why I get a command denied when trying to select user_registration from the user table on enwiki? It use to work just fine [17:07:33] [bz] (8NEW - created by: 2Tim Landscheidt, priority: 4Unprioritized - 6normal) [Bug 52527] tools-webserver-01 seems to be out of memory from time to time - https://bugzilla.wikimedia.org/show_bug.cgi?id=52527 [18:55:38] (03CR) 10Tim Landscheidt: "Memo to self:" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/76313 (owner: 10Platonides) [19:26:56] so [19:27:00] I'm not actually travelling [19:27:05] apparently my flight is tomorrow [19:31:03] YuviPanda|Travel: Advantage of a proper sleep cycle: You can distinguish today and tomorrow :-). [19:31:14] yeah, true [19:31:17] apparently I can't [19:39:33] scfc_de: on the plus side, at least I can fix up some of my bots today [20:02:34] addshore: Do you have admin rights on wikitech? Can you block User:Larygloria01, please? [20:04:44] scfc_de: y u no have admin there? [20:07:46] YuviPanda: Not necessary/too much to screw up? Though it would be nice if we could isolate the block right like the shellmanager stuff. [20:07:53] yeah [20:07:58] * YuviPanda blocks scfc_de [20:08:37] scfc_de: any luck with the Redis things? [20:08:46] by 'things' of course, I mean stats for ganglia [20:10:54] YuviPanda: Not yet; should I take the "raw" stats names ("connected_clients") or rephrase that ("Number of connected clients")? Suggestions for group name? [20:11:09] take it raw, so it is more googleable [20:11:16] what is 'group name'? [20:12:49] http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=tools&h=tools-master&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 => "SGE metrics" [20:15:06] ah [20:15:09] 'Redis metrics'? [20:15:45] Heloo [20:16:03] Hiya scfe_de [20:16:30] What are they doing? :0 [20:20:38] YuviPanda: "db0:keys=922,expires=907", "db9:keys=1,expires=0" => "keys"? Other items to pass through: "connected_clients", "used_memory"? [20:20:38] [20:21:07] scfc_de: let me look, moment [20:21:37] scfc_de: blocked_clients [20:21:55] pubsub_channels:0 [20:24:42] YuviPanda: pubsub is the queue mechanism? [20:25:05] scfc_de: it's a non-reliable (but much faster!) queueing mechanism of sorts that we don't really use now [20:25:12] but Damianz was looking at it yesterday [20:25:16] we just use normal queues [20:25:35] err [20:25:36] normal lists [20:25:37] as queues [20:25:41] since they have an LPUSH and RPOP [20:26:59] Hah, redis-cli outputs \r\n as newlines. [20:27:52] :D [20:27:59] wrap it around with a python script? :P [20:28:16] I need to use a native Perl Redis module in the long term anyhow. [20:29:21] scfc_de: the one that comes with the system is apparently a year and half old :P [20:30:27] I'm older, and still functioning quite alright :-). [20:34:58] :P [20:35:14] scfc_de: but no, there were some changes to the redis protocol over time. they're still backwards compatible (for now) [20:35:16] YuviPanda: There we go: http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=tools&h=tools-redis&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [20:35:26] wooo [20:35:27] ncie [20:35:48] I'll set up as a minutely cron job. [20:35:53] scfc_de: how does it already have stats for 'last hour'?! [20:36:03] ah [20:36:04] nevermind [20:36:34] scfc_de: mail out to labs-l when you're done :) [20:36:45] YuviPanda: I'm done :-). [20:37:25] scfc_de: also where is this configured? when we bring up another instance of redis, will this automatically tag along? [20:38:55] YuviPanda: Oh, you mean *that* "done" :-). No, it still needs to be puppetized. Take a look at /home/scfc/bin/redisstats. For memcached, I think ops preferred a more Pythonic monster. This is just quick & dirty. [20:40:20] YuviPanda: If you look at operations/puppet, ganglia/plugins/memcached.py and ganglia/plugins/memcached.pyconf need to be ported to Redis probably. [20:41:53] scfc_de: yeah, but isn't there something there arleady? since we already have redis in proudction [20:42:31] * YuviPanda reads those files [20:42:53] YuviPanda: Well, I could have searched for modules/redis/files/ganglia/redis.py before I wrote that thing ... *Argl* :-) [20:43:08] hahaha :D :D [20:43:35] So why wasn't it enabled in the first place? [20:43:54] is it automatically enabled? [20:45:17] I see class "redis::ganglia", but it needs to manually included it seems if you look at manifests/site.pp. [20:46:21] So my assumption would be that we need to "include redis::ganglia" in modules/toollabs/manifests/redis.pp. I'll try that out later on toolsbeta. [20:46:28] okay! [20:46:35] (Well, or you could there! :-)) [20:47:57] scfc_de: do we have a toolsbeta-redis? [20:49:06] YuviPanda: For testing, I usually take toolsbeta-puppettestbed and use it for whatever I need to do. [20:49:36] bbiab [20:49:37] scfc_de: Ryan +2'd the disable MONITOR change. once that gets applied we need to restart redis [20:49:44] addshore: redis restart gonna happen in a bit [20:50:14] :0 [20:50:32] No idea what would happen to my jobs ;p [20:50:38] But don't worry :) [20:51:33] :P [21:02:23] 'rename-command MONITOR ""' arrived, okay to restart? [21:05:51] Why does "service redis-server status" say "redis-server is not running" when it is? Hmmm. [21:07:10] scfc_de: perhaps it is just 'redis'? [21:07:10] * YuviPanda isn't sure [21:07:10] scfc_de: yeah, do restart! [21:07:33] Because I'm not root. *Argl* "sudo service redis-server status" = "redis-server is running" [21:07:38] Okay, restarting. [21:07:48] :P [21:07:55] That should have been it. All keys lost? [21:08:00] scfc_de: err, no? [21:08:09] scfc_de: at *worst*, 1 key lost :) [21:08:15] that's the guarantee that redis makes [21:08:40] YuviPanda: Oh, I thought it was memory only. [21:08:59] scfc_de: it uses an append-only-file to store data, so it recovers across restarts [21:09:00] But yeah, "uptime_in_seconds:62" = restart successful. [21:09:14] scfc_de: however, the total amount of data is limited to the memory size [21:09:25] scfc_de: so the file is used solely as a means to recover from crashes [21:09:42] Is the file rotated? Or does it grow indefinitely? :-) [21:09:49] it is 'compacted' [21:09:51] from time to time [21:09:59] as it reaches a specified sized [21:10:02] *size [21:10:09] where it kills out keys that aren't there [21:10:17] so the file can't grow larger than the size of the memory :P [21:11:32] Ah, okay. Re Ganglia, different approach: I'll add temporarily "redis::ganglia" as a selectable class to the wikitech UI, select that, and see what happens. [21:12:15] alright! [21:14:07] YuviPanda: Doesn't work: "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Redis] is already defined; cannot redefine at /etc/puppet/modules/toollabs/manifests/redis.pp:40 on node i-00000876.pmtpa.wmflabs". So I'll need to test it on toolsbeta. [21:14:16] heh :P [21:14:18] ok [21:28:15] [bz] (8NEW - created by: 2Tyler Romeo, priority: 4Normal - 6enhancement) [Bug 52354] Run Minion testing instance for security testing - https://bugzilla.wikimedia.org/show_bug.cgi?id=52354 [21:30:03] YuviPanda: I don't quite get. I included redis::ganglia on toolsbeta-puppettestbed, yet the Ganglia graph stays the same. Looked at site.pp, saw mc[0-9][0-9] are running that as well, but can't find any host with that name on ganglia.*wikimedia*.org and Redis stats. Hmmm. [21:30:32] hmm, ganglia is itself mostly a mysterious black box to me [21:32:47] Ah, okay, found a "Redis eqiad" group (which I don't see when I look for "memcached" :-)). Let's see what's special about rdb1001 http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Redis+eqiad&h=rdb1001.eqiad.wmnet&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [21:41:11] *ping* addshore [21:46:15] [bz] (8NEW - created by: 2Steinsplitter, priority: 4Unprioritized - 6normal) [Bug 52534] Special:UserLogin - https://bugzilla.wikimedia.org/show_bug.cgi?id=52534 [21:46:45] (03PS1) 10Yuvipanda: Remove receiver.py, has been moved to gerrit-to-redis project [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77641 [21:47:10] (03PS1) 10Yuvipanda: Add .gitreview file [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77642 [21:47:19] (03CR) 10Yuvipanda: [C: 032 V: 032] Remove receiver.py, has been moved to gerrit-to-redis project [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77641 (owner: 10Yuvipanda) [21:47:30] (03CR) 10Yuvipanda: [C: 032 V: 032] Add .gitreview file [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77642 (owner: 10Yuvipanda) [21:51:15] YuviPanda: Ganglia even has a logfile! "[PYTHON] Can't find the metric_init function in the python module [redis].", "Unable to find the metric information for 'connected_clients'. Possible that the module has not been loaded.", "Unable to find the metric information for 'connected_slaves'. Possible that the module has not been loaded.", etc. Let's see ... [21:51:26] ah! [21:51:34] perhaps it's looking at a wrong file? [21:51:38] do things have to be cloned, perhaps? [21:51:50] sorry, I don't think I've any idea what I'm talking about [21:52:35] YuviPanda: Me neither. Complicated construct, Ganglia. [21:52:39] heh [21:54:35] redis.py is in /usr/lib/ganglia/python_modules, so that looks alright. Hmmm. [21:58:34] scfc_de: wtf?! OperationalError: (2006, 'MySQL server has gone away') [22:00:07] whelp [22:03:11] grr [22:06:27] (03CR) 10Yuvipanda: "Testing gerrit-to-redis!" [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77642 (owner: 10Yuvipanda) [22:06:31] tada! [22:06:36] whoopydeewhoop! [22:06:39] worksie! [22:06:41] nice :) [22:08:02] scfc_de: migrated grrrit-wm to tools-redis :) [22:08:49] (03CR) 10Yuvipanda: "Finally migrated to tools-redis, woohoo! :)" [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/77642 (owner: 10Yuvipanda) [22:09:11] scfc_de: but this will add only like, 2-3 entries at most to the keyspace, so won't show much in terms of growth [22:13:18] YuviPanda: Nice! [22:13:34] tools-db has inserts that are really, *really* slow [22:13:35] sigh [22:13:58] scfc_de: so this is now running as a 'registrar' of sorts. you run a command on your client, as your tool, and that lets you 'subscribe' to all gerrit changesets in real time [22:13:58] :) [22:14:00] soon all github too [22:14:02] i need to write docs [22:14:03] for thise [22:14:11] scfc_de: ooh, also - i am using zeromq now :D [22:14:56] I mean, it is just regular ol' network communications, and I could've just as easily used regular sockets [22:15:41] I should go sleep now :) [22:15:42] gnite [22:24:40] YuviPanda: Good night! Don't miss your flight :-). [22:24:50] scfc_de: hehe, :P it's still ~22 hours away [22:25:47] scfc_de: are you using puppetmaster::self? [22:33:45] YuviPanda: For toolsbeta-puppettestbed? Yes. [22:33:54] hmm, so I just add that role [22:33:58] and force a run [22:34:04] and then magically I'll find the puppet repo contents [22:34:05] there [22:34:11] as a git repo? [22:36:34] scfc_de: also, we need a redis-cli on exec_environ. Thoughts on how to do that? [22:36:49] scfc_de: since it is in the 'redis' package and we can't get that without getting the server too? [22:37:31] hmm, perhaps I could write a trivial service in Go that does that :D [22:37:39] Redis over the web! [22:38:10] bwahahahaha! :P [22:41:43] YuviPanda: puppetmaster::self is documented ... https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster - but you are basically right. [22:45:08] YuviPanda: Re redis-cli, we could package it privately (conflicting to redis-server) and install it on exec_environ, but personally I would shy away from that unless there is wider interest. Is the redis protocol binary? "nc 127.0.0.1 6379" and "INFO" doesn't work. [22:45:23] I think it is binary, yeah [22:45:25] (not too sure) [22:46:07] YuviPanda: Ah, no! redis doesn't just output \r\n, it requires that for input as well. Try "nc -C tools-redis 6379". [22:46:12] (And INFO, etc.) [22:46:46] ah [22:46:47] nice! [22:46:58] hehe, I just tried MONITOR :P [22:47:16] scfc_de: can you add that to tools-help? [22:47:32] "For debugging, [..."]? [22:49:05] yeah [22:50:25] scfc_de: I added the role, forced a puppet run, nothing in /var/lib/git [22:51:02] Which host? [22:51:02] /etc/puppet/manifests is also empty [22:51:12] proxy-project instance in project project-proxy [22:51:19] (:D) [22:51:44] Was there any changes in the Puppet run? [22:52:49] npe [22:52:52] otice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully [22:52:52] notice: /Stage[main]/Base::Puppet/Exec[puppet snmp trap]/returns: executed successfully [22:52:52] notice: Finished catalog run in 16.76 seconds [22:54:13] !log deployment-prep platonides: --help [22:54:16] Logged the message, Master [22:55:17] YuviPanda: redis: https://wikitech.wikimedia.org/w/index.php?&diff=79783 [22:55:43] YuviPanda: puppetmaster worked for me without glitch so far. [22:55:45] curious log message there :P [22:55:56] scfc_de: sigh. I guess I'll just head to sleep and try again at wikimania [22:56:10] yeah, good enough. thanks scfc_de [22:56:30] !log deployment-prep platonides: git reset --hard to restore /data/project/apache/common-local/php-master/extensions/Translate/specials/SpecialManageGroups.php (bug 52534) [22:56:32] Logged the message, Master [22:56:46] Probably "log --help" :-). Good night (again), YuviPanda! [22:56:55] :D [22:57:01] let's hope I actually sleep this time ;) [22:57:08] !log deployment-prep platonides: test [22:57:11] Logged the message, Master [22:58:03] Let's hope you fly today :-). [22:58:45] !log [22:58:46] Message missing. Nothing logged. [22:59:49] !log foo [22:59:50] Message missing. Nothing logged. [22:59:59] !log foo [23:00:00] Message missing. Nothing logged. [23:00:47] !log foo [23:00:48] Message missing. Nothing logged. [23:01:51] <{{Guy}}> Why is wm-bot trying to log stuff? [23:02:53] {{Guy}}: I think Platonides is using the command line logging tool that causes wm-bot to say "!log", but syntax errors along the way ... [23:03:41] <{{Guy}}> Ahh. Is there a ticket in for that? petan should probably fix that. [23:09:02] {{Guy}}: No, feel free. TBH, I don't know if the "Message missing. Nothing logged." bit gets replayed to the command line. [23:09:21] &ping [23:09:22] Pinging all local filesystems, hold on [23:09:23] Written and deleted 4 bytes on /tmp in 00:00:00.0011600 [23:09:24] Written and deleted 4 bytes on /data/project in 00:00:00.0073210 [23:09:38] <{{Guy}}> &trusted [23:09:38] I trust: test (2trusted), [23:09:50] <{{Guy}}> lol [23:10:19] <{{Guy}}> &whoami [23:10:20] You are unknown to me :) [23:10:38] <{{Guy}}> I don't know you either wm-bota... [23:10:55] <{{Guy}}> wm-bot is my buddy on the other hand. [23:11:07] <{{Guy}}> wm-bot: thanks. [23:11:07] Hey {{Guy}}, you are welcome! [23:11:08] Hi {{Guy}}, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [23:50:38] !log tools-beta Added scfc_de to local-admin [23:50:39] tools-beta is not a valid project. [23:50:56] !log toolsbeta Added scfc_de to local-admin so I don't log myself out again :-) [23:50:57] Logged the message, Master