[08:04:24] <matanya>	 !log NFS Issues: cannot open directory .: Stale NFS file handle
[08:05:18] <matanya>	 Coren: ^
[09:38:03] <kaldari>	 Coren, petan: ToolLabs seems to be down. Can't even ssh to it.
[09:39:41] <matanya>	 kaldari: it is
[09:39:46] <matanya>	 labstore4 is dead
[09:39:56] <matanya>	 see admin log
[09:52:45] <petan>	 kaldari: hi
[09:53:01] <kaldari>	 howdy
[09:53:50] <petan>	 this needs someone from ops to fix it I don't have access to nfs server
[09:54:02] <matanya>	 petan: jump to -operaions
[10:08:13] <Leloiandudu>	 hi guys. is tools-login.wmflabs.org down? i get conn refused via ssh
[10:08:30] <matanya>	 yes Leloiandudu 
[10:11:42] <GerardM->	 http://tools.wmflabs.org/reasonator does not do https but is down as well
[10:12:21] <Leloiandudu>	 thanks, matanya
[11:23:01] <prolineserver>	 I'm getting "Bad Request Your browser sent a request that this server could not understand." when I try to access https://tools.wmflabs.org/
[11:23:38] <prolineserver>	 And I'm kicked off from tools-login.wmflabs.org immediately
[11:42:01] <GerardM->	 any news on the availability of labs functionality ... as it is I am about to cancel a demo
[12:04:09] <fluff>	 can't login to tools-login, any known problems not in topic?
[12:06:46] <prolineserver>	 fluff:  GerardM-: http://lists.wikimedia.org/pipermail/labs-l/2014-February/002128.html
[12:07:38] <fluff>	 thanks prolineserver 
[12:11:28] <Blahma>	 Thanks from me too. Any idea who "Mark-Andre" is and when he will be available to fix the problem? Insider WMF communication tends to be quite cryptic at times...
[12:14:16] <prolineserver>	 Blahma: https://meta.wikimedia.org/wiki/User:MPelletier_(WMF)
[12:14:42] <prolineserver>	 ping Coren
[12:16:39] <Blahma>	 prolineserver: Thanks. Let's hope he'll be able to fix it today. I realize that it's early Sunday morning in the US, but here in Europe I'm getting complaints from users who cannot create new articles because my tool is down :-(
[12:20:10] <GerardM->	 At Reasonator we have over 4000 page views ...
[12:20:15] <GerardM->	 a dat
[12:20:20] <GerardM->	 day
[12:23:38] <Blahma>	 GerardM-: My users produce some 50 articles a day, which is much less, but it is even more crucial for me to preserve their loyalty because they are few.
[12:28:03] <Steinsplitter>	 toolos down?
[12:29:19] <fluff>	 Steinsplitter: yes
[12:29:28] <fluff>	 Steinsplitter: http://lists.wikimedia.org/pipermail/labs-l/2014-February/002128.html
[12:30:05] <GerardM->	 Blahma there is no competition in misery
[12:30:23] <GerardM->	 50 aricles in what language ?
[12:30:48] <Steinsplitter>	 fluff: thx :)
[12:36:11] <Blahma>	 GerardM-: Mine is a tool that combines Parsoid's stripped HTML with machine translation and back to wikitext, which provides very good quality for Czech to Slovak (and vice versa) including some specially crafted "template translations" etc.
[12:45:18] <GerardM->	 Blahma does it have some documentation somewhere  ?
[12:46:41] <Blahma>	 Not yet. I'm going to write a WMF blog post about it sometime soon, but want to settle a few issues first. I also work with Wikidata, btw – use them to "translate" links whenever possible.
[12:57:01] <GerardM->	 Blahma http://ultimategerardm.blogspot.nl/2014/02/reasonator-johann-sebastian-bach-in-so.html a first step of generating text on the fly using Wikidata info
[12:58:26] <Blahma>	 Thank you, that sounds exciting. Will read that while they are fixing this.
[13:37:39] <Steinsplitter>	 sitic: arbeitest du für die wmf?
[13:41:45] <sitic>	 Steinsplitter: äh nein, hatte es nur die letzten Tage mitbekommen (ist jetzt schon das dritte mal diese Woche, nach meiner Zählung)
[13:42:02] <Steinsplitter>	 ah
[14:57:35] <benestar>	 tools project is down again
[14:57:40] <benestar>	 this really suks
[14:57:42] <benestar>	 *sucks
[14:59:12] <GerardM->	 So what can be done ?
[14:59:26] <GerardM->	 it is a matter of resourcing
[14:59:28] <benestar>	 I don't know
[14:59:34] <benestar>	 but it really really sucks
[14:59:38] <GerardM->	 it is now concentrated in too few people
[14:59:54] <benestar>	 I thought the tools project would be stable and bots not
[15:00:05] <benestar>	 obviously it isn't the case :/
[15:00:31] <GerardM->	 it is MORE stable than the previous iteration
[15:00:56] <prolineserver>	 Well, complain to the responsible people at WMF ;)
[15:00:56] <benestar>	 but there are much more people using it I think
[15:01:08] <valhallasw>	 GerardM-: more stable than what, exactly?
[15:01:12] <benestar>	 hehe, I think this is a financial question
[15:01:17] <GerardM->	 The German iteration
[15:01:30] <valhallasw>	 Heh. That's what the WMF would like you to believe, yes.
[15:01:38] <GerardM->	 given that there is enough money, it is not
[15:06:54] <Coren>	 Oh, eff.
[15:07:21] <Coren>	 No, it's a question that fixing the problem on the current fileserver would require bringing all of labs down for over a day.
[15:08:47] <benestar>	 Coren: hey :DD
[15:08:57] * Coren  kicks XFS

[15:09:16] <benestar>	 Coren: is there any way I can help you to prevent such outages in future?
[15:10:10] <Coren>	 No.  We're just about to switch to eqiad, and no XFS there though.
[15:12:25] <benestar>	 Coren: I think tools was down quite often in recent weeks...
[15:14:57] <Coren>	 benestar: The bug in the kernel that screws XFS up appears to be increasingly likely with load.  :-(
[15:15:08] <Coren>	 Well, disk activity.
[15:16:03] <prolineserver>	 tools is back, tankts Coren!
[15:16:13] <prolineserver>	 thanks
[15:16:54] <benestar|afk>	 Coren: thanks a lot :)
[15:17:28] <benestar|afk>	 hey, not back for me...
[15:18:13] <matanya>	 thanks Coren 
[15:51:09] <Coren>	 Everyone give a word of prayer for nothing wrong happening so that we can switch to eqiad really soon.  With a bit of luck, we might be able to start moving tools in a week.
[15:53:44] <valhallasw>	 Coren: I don't think grrrit-wm agrees :D
[15:54:21] * Coren  tries to go restart it

[15:54:51] <valhallasw>	 it might be throwing some exception, and then restarting...
[15:56:43] <Coren>	 That seems to have fixed it.
[18:54:17] <Fatemi127>	 Hi, I need to access for  Bot projects in persian wiki  (fa.wikipedia),
[18:55:05] <valhallasw>	 !tools
[18:55:05] <wm-bot>	 There are multiple keys, refine your input: tools-admin, toolsbeta, tools-bug, tools-request, toolsvslabs, tools-web, 
[18:55:10] <valhallasw>	 !tools-request
[18:55:10] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request
[18:55:15] <valhallasw>	 Fatemi127: ^
[18:56:05] <valhallasw>	 !tools-request delete
[18:56:05] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request
[18:56:09] <valhallasw>	 !delete tools-request
[18:56:14] <valhallasw>	 :/
[18:56:15] <valhallasw>	 !help
[18:56:15] <wm-bot>	 !documentation for labs !wm-bot for bot
[18:56:18] <valhallasw>	 !wm-bot
[18:56:18] <wm-bot>	 http://meta.wikimedia.org/wiki/WM-Bot
[18:57:21] <valhallasw>	 !tools-request del
[18:57:22] <wm-bot>	 Successfully removed tools-request
[18:57:26] <valhallasw>	 !tools-request is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access_to_Tool_Labs
[18:57:26] <wm-bot>	 Key was added
[18:57:47] <valhallasw>	 !tools-request is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access_to_Tool_Labs (request form: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request)
[18:57:48] <wm-bot>	 This key already exist - remove it, if you want to change it
[18:57:51] <valhallasw>	 !tools-request del
[18:57:52] <wm-bot>	 Successfully removed tools-request
[18:57:54] <valhallasw>	 !tools-request is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access_to_Tool_Labs (request form: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request)
[18:57:54] <wm-bot>	 Key was added
[19:00:25] <Fatemi127>	  tools-request > Hi, I need to access for  Bot projects in persian wiki  (fa.wikipedia), 
[19:03:46] <valhallasw>	 Fatemi127: follow the instructions at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access_to_Tool_Labs 
[19:04:21] <Fatemi127>	 valhallasw! thanks :)
[19:12:37] <Fatemi127>	 valhallasw: I have shell and I upload SSH public key but now I need to bot tools.
[19:18:20] <scfc_de>	 Fatemi127: You would have needed to follow the instructions at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Requesting_access_to_the_.27tools.27_project (file a request at https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request), but I can add you without one; one moment, please.
[19:18:43] <scfc_de>	 Fatemi127: Oh, you did file a request.  Sorry.
[19:20:28] <scfc_de>	 Fatemi127: I've added you to the Tools project.  If you want to run a bot, please be advised that bots cannot be run on tools-login; you need to run them as a job on the grid.  See https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid for how to do that.
[19:21:39] <valhallasw>	 !RC+ wikitech Nova_Resource:Tools/Access_Request/*
[19:21:46] * valhallasw  looks at wm-bot 

[19:21:51] <valhallasw>	 @RC+ wikitech Nova_Resource:Tools/Access_Request/*
[19:21:52] <wm-bot>	 Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it
[19:24:47] <Krenair>	 @RC+ wikitechwiki Nova_Resource:Tools/Access_Request/*
[19:24:47] <wm-bot>	 Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it
[19:25:06] <scfc_de>	 valhallasw: wikitech's RC isn't fed to irc.wikimedia.org, and wm-bot gets it from there.
[19:25:24] <valhallasw>	 Right. I was checking the irc.wm.o channel list
[19:25:38] <valhallasw>	 which even has the beta projects O-o
[19:27:19] <scfc_de>	 There's an open bug about this; https://bugzilla.wikimedia.org/show_bug.cgi?id=34685.  But I don't know if Andrew either has time to fix this before the eqiad move or if he should :-).
[19:30:56] <valhallasw>	 Right. It's not too important.
[19:33:25] <valhallasw>	 ok, opened a bug that depends on 34685 to not forget about this
[19:46:49] <petan>	 valhallasw: that bug is open for years, nobody cares about this
[19:50:47] <valhallasw>	 petan: well, I guess this is a bump then :-)