[01:05:03] For the last hour I got a crap ton of failed to send emails from crom [01:05:11] *cron [01:08:20] Betacommand: I'm cleaning up some clogged queues; you should receive 31 more mails if I'm correct. [01:08:48] scfc_de: why was I getting an undeliverable error? [01:11:36] Betacommand: The basic problem was that mail from jsub was addressed to $user@$host.pmtpa.wmflabs, while it should have been to $user@tools.wmflabs.org. [01:26:35] !log tools chmod g-w ~{bgwhite,daniel,euku,fale,henna,hydriz,lfaraone}/.forward (test: sudo find /home -mindepth 2 -maxdepth 2 -type f -name .forward -perm /g=w -ls) [01:26:37] Logged the message, Master [01:36:50] So http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page can't be accessed from within Labs. [01:37:07] Is there an internal hostname that can be accessed? [01:47:44] superm401: That host resolves to 208.80.153.219 which in the instance list of the Beta project is listed as the public IP for deployment-cache-text1. Could you try that? (You may have to fake a "Host:" header.) [01:50:00] Thanks, scfc_de, we'll try that. [01:53:31] Hello, everyone [01:53:55] does anyone here use a versioning system of some sort? [01:54:41] I am getting tired of copying all of my files over by hand (and making sure I don't break anything) and having to run take (and possibly having it fail) then having to change the permissions [02:05:02] Magog_the_Ogre: I think many people use Git or something similar. [02:05:53] thanks scfc_de [02:06:14] I am Linux, therefore I do git on home machine [03:09:16] Anyone know who JaGa is? They have a tool on labs (dplbot) I want to know more about. [03:10:19] Also is Tim Starling still in charge of updating Special:UncategorizedPages on enwp? [03:15:55] T13|sleeps: Wasn't there a big relaunch of running some special page lists recently? [03:16:31] No idea. Page says last updated on the 28th [03:17:07] Apparently there's a bug I'm trying to track where pages only in hidden cats are showing up uncategorized [03:17:26] T13|sleeps: Re dplbot, https://wikitech.wikimedia.org/wiki/User:JaGa & ff. not enough? [03:18:32] ff? [03:19:28] There's only 100-130 pages on list... not sure why it can't be updated more often. [03:19:54] How is it updated anyways? [03:20:11] scfc_de: I think Nemo_bis would know more about the 'run special pages now and then' thing [03:20:12] ff. = https://en.wikipedia.org/wiki/User:DPL_bot, https://en.wikipedia.org/wiki/User_talk:JaGa [03:20:30] YuviPanda: That's who I had in mind :-). Looking for the bug. [03:20:43] Oh. I left a message there already. Was hoping to find him on IRC [03:22:21] I think I meant https://bugzilla.wikimedia.org/show_bug.cgi?id=15434, but it's hard to tell if that's the one. [03:27:59] There are some cron jobs in Puppet, manifests/misc/maintenance.pp, but that doesn't look enabled. And just as I wanted to advise that -operations probably has more knowledge on this, I see you're way ahead of me :-). [03:43:40] !log tools Cleaned up all exim queues [03:43:41] Logged the message, Master [03:45:21] ... [03:45:37] scfc_de: I have 125 "Mail delivery failed" emails sitting in my spam folder >.> [03:46:41] but thanks for doing it :D [03:50:42] Well, I could have done an extensive study on who would want which kind of stuck mail and then crafted some elaborate filter, but it already took long enough to do it this way :-). [03:52:34] We need to get Icinga for Labs running reliably and then enable checks like https://bugzilla.wikimedia.org/58871 so that stuck queues are detected *early* :-). [03:53:56] I expect a trouting for suggesting such a thing.. lol [05:13:01] Any labs ops here? Not sure what's going on but my cvn instances are unreachable. [05:13:23] Tried restarting and ssh (both via ProxyCommand and directly from ssh bastion), no response. [05:13:29] I can ping it from bastion1 though [05:13:36] and ganglia also shows some activity [05:13:50] https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn [05:14:04] The web server is unreachable though http://cvn.wmflabs.org/api.php [05:14:08] and ssh unresponsive [05:14:33] network completely dissappeared [05:14:33] http://ganglia.wmflabs.org/latest/graph.php?r=week&z=xlarge&c=cvn&m=load_one&s=by+name&mc=2&g=network_report [05:14:42] http://ganglia.wmflabs.org/latest/graph.php?r=week&z=xlarge&c=cvn&g=network_report [05:15:27] Coren's still travelling, andrewbogott's afk, I don't know if Ryan_Lane's free :-). [05:17:13] My guess (ssh accepting connections, but then hanging): GlusterFS is somehow hanging so ssh can't verify keys. [08:10:23] Krinkle|detached: I fixed logins on cvn (scfc_de was right about the problem.) Let me know if other bits are still broken... [12:59:47] There are two public databases, which have weird names: `p50380g50841__deployed_without_password_that_is_p` and `p50380g50863__projects_should_have_passwords_p`... [14:28:29] anyone able to set a command in wm-bot ? [14:31:03] Betacommand: ?? [14:31:07] @whoami [14:31:07] You are root identified by name .*@wikimedia/Technical-13 [14:31:23] @whoami [14:31:23] You are admin identified by name .*@wikipedia/Cyberpower678 [14:41:58] Betacommand: I can [14:52:44] petan|wk: already done [15:03:10] @whoami [15:03:10] You are trusted identified by name .*@wikimedia/.* [15:33:37] xtools has overloaded again. [15:33:43] Frr. [15:33:47] Gr.. [15:33:49] Grrr. [21:57:38] Hello everyone. Has anyone on here installed git [21:57:44] as a way to manage their code on labs? [22:00:08] Magog_the_Ogre: I'm using it for various projects on Tool Labs. [22:00:44] But I'm guessing you have a question to ask ;-) It's best to just ask it, instead of first finding someone who might be able to answer it. [22:04:36] heh [22:04:39] I guess this is a geek den [22:04:43] I don't need to explain myself [22:04:59] we don't bother with any of those nice social queues [22:05:09] (oh wait, not all geeks are as oblivious as I am, shoot) [22:05:11] ANYWAY [22:05:17] * valhallasw shoots [22:05:35] I would like to set up git for my projects [22:06:15] because it's a huge pain in the rump to transfer from my normal user into the web user, run take, change permissions, fix further problems, etc. [22:06:46] I'd rather the code not be displayed to the public, because I wrote half of it before I had a real programming job, so it's ugly beyond all reason [22:07:01] I run a Linux machine, so I'm capable of running the git server myself if necessary [22:07:16] OK. You develop locally? [22:07:43] yes [22:07:58] And git would basically be the deploy tool. Ok. That is doable. [22:08:06] correct [22:08:14] Let me try something first... [22:08:40] alternatively, I could develop locally, and then upload to git, and d/l on the server [22:10:59] OK. I think the easiest is the following. Create a bitbucket account (which allows free, private git repositories) [22:11:19] Create a git repository on your local code base, push that to bitbucket [22:11:36] Clone the bitbucket repo as your tool user [22:12:13] then if you make changes at home, git commit && git push; then login to labs and git pull [22:12:40] that's the workflow I use for several projects (albeit with github as intermediary instead of bitbucket) [22:14:19] and bitbucket is better because it doesn't publish my code? [22:14:44] correct [22:15:05] github's interface is nicer, and it has a larger community, but no free private repositories [22:15:24] very cool; thanks [22:15:45] I always feel like an idiot when I move to a new software platform [22:19:13] Magog_the_Ogre: I would also encourage you to suppress your shame at the condition of the code and put your repositories out where people can see them. Writing "utilitarian" code isn't a sin but hiding open source code from the world is. :) [22:24:48] It's also the only way to improve your code; if there's nobody telling you what's wrong, how can you learn what's right? [22:27:59] I've been writing code professionally for … a long time … and I almost always cringe when I look at any code I wrote more than 6 months ago. We are all learning new things all the time and we are all at some level ashamed of the sloppy/ugly/inelegant code we just finished writing. [22:36:20] bd808, scfc_de, there is a matter of degrees here [22:36:30] I literally didn't even use objects in my code [22:36:45] I used freaking goto's. I am not kidding. [22:37:12] Does the code do something cool and/or useful? [22:37:13] This isn't just ugly. It's code which is so hideous to behold, it would make Medussa curl up in a ball and die. [22:38:02] Publish it and some internet samaritan may send you a pull request that cleans it all up :) [22:41:00] Just as a random example: https://github.com/rashidkpc/Kibana/pull/4