[01:03:09] <TParis>	 Any tools lab admins on?
[01:17:34] <TParis>	 ping Coren
[02:01:52] <wikibugs>	 3Tool Labs tools / 3X!'s tools: Xtools offline - 10https://bugzilla.wikimedia.org/72104#c11 (10TParis) I have looked into this.  The file executes successfully on the server so there is no reason it shouldn't operate on the web.  However, I have no rights to the /public_html/ folder so I cannot do further te...
[04:59:39] <Coren>	 TParis: Sorry, was out on the town.  :-)  Something I can do to help you?
[06:29:50] <TParis>	 Coren: I'll catch you tomorrow, thanks anyway
[06:30:19] <TParis>	 If you have time and the inclination though, something is wrong with xtools, Cyberpower is on vaca, and I dont have write permissions to the public_html folder to test anything out.
[06:30:27] <TParis>	 Have to go, though, gnight
[08:56:56] <gifti>	 Coren: fyi https://tools.wmflabs.org/catscan2/notice.html
[11:02:14] <gifti>	 coren, petan: can you approve a new tools membership?
[15:17:40] <GerardM->	 Hoi I am increasingly upset that I cannot use the tools that I depend on
[15:18:27] <GerardM->	 To be honest, as far as I am concerned I am not willing to allow for finger pointing
[15:19:40] <GerardM->	 it is now multiple days that I cannot do my thing
[15:19:45] <GerardM->	 I am upset
[15:20:08] <gifti>	 what is this caused by?
[15:20:36] <GerardM->	 As far as I am aware a lack of resources given to necessary webservices
[15:20:45] <GerardM->	 http://tools.wmflabs.org/autolist/index.php
[15:21:01] <GerardM->	 http://tools.wmflabs.org/toolscript/index.html?pastebin=guXqaGQE is another
[15:21:41] <GerardM->	 we know why these services take their memory and we know that giving more memory works
[15:22:05] <GerardM->	 we also know them to be stable in what they do
[16:11:37] <Coren>	 GerardM-: Except that it's not.  autolist is crashing repeatedly because it has an error in its code (array_chunk() expects parameter 1 to be array, boolean given in /data/project/autolist/public_html/index.php on line 347)
[16:12:59] <GerardM->	 thanks for the info
[16:13:04] <GerardM->	 I relayed it to Magnus
[16:13:11] <Coren>	 toolscript is up now, but "PHP Notice:  Undefined index: items in /data/project/toolscript/public_html/misc.php on line 71"
[16:14:21] <Coren>	 So more memory will help nothing, I'm afraid.
[16:15:10] <GerardM->	 well the last time we spoke it did
[16:15:23] <GerardM->	 and I am upset
[16:15:27] <Coren>	 What Magnus needs most, I think, is help.  He's only human, and - like every volunteer - has no infinite time.
[16:15:50] <GerardM->	 I know
[16:16:11] <GerardM->	 he has a life ... and truthfully his day job is more relevant than what he does for us
[16:16:36] <GerardM->	 he works in malaria research
[16:16:43] <Coren>	 Indeed.
[16:17:17] <Coren>	 Trust me, Gerard, if I had the ability to do so I'd just assign a dev to help him.
[16:17:52] <GerardM->	 <grin> I do but there is only so much what you can do
[16:18:40] <Coren>	 The single most important thing he could do is add maintainers to his tools so that others can help when something goes wrong.  Most of his tools have only him listed as maintainer.  :-(
[16:19:15] <Coren>	 TParis tried to help last night, but was unable to do so because he did not have write permission.
[16:19:42] <gifti>	 Coren: i have asked before, but have you seen http://tools.wmflabs.org/catscan2/notice.html?
[16:20:13] <gifti>	 is there anything you can do?
[16:20:43] <Coren>	 I have, gifti, and I wasn't the only one to notice - so did WMF engineering bosses.  There's nothing *I* can do, but maybe they'll be able to assign someone to help Magnus with coding.
[16:20:59] <gifti>	 mh, ok
[16:21:21] <Coren>	 I did tell them how valuable catscan is, fundamentally, and that it needs development help to stabilize it.
[16:21:59] <Coren>	 And that we can't expect Magnus to singlehandedly keep the word running.  Its unfair to /him/ as much as it's unfair to his users.
[17:51:40] <physikerwelt__>	 hi I have problems accessing this instance https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000432.eqiad.wmflabs
[17:51:57] <physikerwelt__>	 all other instances I tried work fine
[17:52:09] <physikerwelt__>	 can someone try to access it
[17:52:52] <physikerwelt__>	 the status information do not seem to be good either http://icinga.wmflabs.org/cgi-bin/icinga/extinfo.cgi?type=1&host=mlp.eqiad.wmflabs
[17:59:14] <Coren>	 physikerwelt__: Lemme take a quick look.
[17:59:31] <physikerwelt__>	 Coren: thank you
[18:03:41] <Coren>	 physikerwelt__: It looks quite down.
[18:03:59] <physikerwelt__>	 I tried to reboot it
[18:04:13] <physikerwelt__>	 but that did not help
[18:04:37] <Coren>	 Indeed, it doesn't even seem to be able to reboot.
[18:06:09] <Coren>	 mws doesn't seem all that healthy either (root filesystem seems to have issues)
[18:07:27] <Coren>	 Everything else seems to be happy, though.  Lemme look under the hood.
[18:07:36] <physikerwelt__>	 yes. But I could simply build a new mws
[18:07:47] <Coren>	 not so with mlp?
[18:08:16] <physikerwelt__>	 It would require some more work... there is some data in the database
[18:10:31] <physikerwelt__>	 but if it's too complicated to restore I can do that
[18:17:41] <Coren>	 Lemme try to see if I can use an untested method to rescue them.  It may or may not work, but it's a good case study
[18:19:53] <wikibugs>	 3Tool Labs tools / 3X!'s tools: Xtools offline - 10https://bugzilla.wikimedia.org/72104#c12 (10Helder) (In reply to Cyberpower678 from comment #2) > I'm on a Wikibreak right now.  Use https://tools.wmflabs.org/supercount for > the edit counter.  Also you'll get more attention if you reported the issue > on h...
[18:21:39] <gifti>	 Coren: is it allowed to use a non-free programming language on labs (compile at home, execute on labs)?
[18:22:07] <Coren>	 gifti: It's...  an odd gray area.  What language are you thinking of?
[18:22:13] <gifti>	 the source code would be free, but it is not usable by others then :\
[18:22:15] <gifti>	 purebasic
[18:22:51] <Coren>	 I'd ask Legal.
[18:23:37] <Coren>	 That's an odd edge case, really.  Does it mean it's impossible to have an open source Effeil program for instance?
[18:27:17] <gifti>	 aren't all eiffel compilers open source?
[18:27:54] <Coren>	 gifti: Maybe now.  The last time I checked (admitedly, some 23 years ago) there was just the one commercial one.  :-)
[18:28:05] <gifti>	 :D
[18:28:33] <gifti>	 so, will you ask legal, or should the use in question file a bug?
[18:28:36] <gifti>	 *user
[18:29:13] <Coren>	 The user should probably file a bug.
[18:29:30] <Coren>	 Legal may have questions and playing telephone is !fun.  :_)
[18:29:38] <gifti>	 hehe
[18:30:11] <gifti>	 Coren: could you please approve https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Inkowik?
[18:32:11] <gifti>	 Coren: is there a bugzilla component for legal or is it better to write an email?
[18:32:48] <Coren>	 Hm.  THere is no component for legal afaik; open a general labs bug and email legal about it?
[18:33:10] <gifti>	 ok
[18:39:50] <Coren>	 physikerwelt__: Yeah, the method shows promise but won't work without extra tweaking.  Lemme try one last thing for mlp
[18:44:11] <physikerwelt__>	 Coren cool
[18:53:31] <Coren>	 physikerwelt__: Yeah, sorry, I won't be able to help recover them.  The "good" side is that with the new Openstack release we will have a good rescue method but it'll require support in the images.
[18:54:30] <Coren>	 I'll put "tweak images to make them rescuable" at the top of next week's todo list.  But it won't be retroactive.  :-(
[18:55:19] <Coren>	 So those instances are lost.
[18:56:17] <physikerwelt__>	 Coren: Thank you for your effort
[18:57:24] <physikerwelt__>	 I'll create new instances and it will be fine...
[18:57:55] <physikerwelt__>	 if that helps the thing that was abnormal on the mlp instance was a modified mariadb data directory
[18:57:56] <physikerwelt__>	 but I can not imagine how this could be problematic
[18:59:31] <Coren>	 I don't see how that could have been it.
[19:00:11] <Coren>	 But both of those instances live on a host that had crashed badly last week and caused a lot of issues.
[19:00:29] <Coren>	 That may well be the root cause and nothing from within the instances themselves.
[19:01:31] <physikerwelt__>	 the diagrams suggested that the problem occured after a failed puppet run
[19:01:46] <physikerwelt__>	 https://tools.wmflabs.org/nagf/?project=math#h_mlp_memory
[19:02:19] <Coren>	 I should expect it more likely that the failed puppet run was the symtop and not the cause.
[19:02:52] <Coren>	 But the timing on the 8th, clearly point to when virt1005 asplode.
[19:03:21] <physikerwelt__>	 as you can see after one puppet run the load goes up and two days later mlp is offline
[19:03:50] <physikerwelt__>	 ok I did not know that
[19:03:51] <physikerwelt__>	 however, thanks for your support
[19:03:55] <Coren>	 Possibly two independent issues.
[19:04:00] <physikerwelt__>	 I have delete those instances and create new instances tomorrow
[19:04:16] <Coren>	 Allright; sorry I couldn't be of more help.
[19:04:17] <physikerwelt__>	 Yes. At lest that's what the statics indicate
[19:05:11] <physikerwelt__>	 no it's my fault. I should have created an sql dump
[21:36:31] <gifti>	 Coren: does catscan2 have any different limits than normal tools?
[21:37:20] <Coren>	 Yes, it has almost twice the memory allocation.
[21:37:42] <gifti>	 can you give that to catscan3 as well?
[21:37:44] <Coren>	 Well, twice the /vmem/ allocation, that's about three times the amount of actual physical ram.
[21:38:09] <Coren>	 Sure.  Experimental version?
[21:38:13] <gifti>	 yes
[21:38:34] <Coren>	 Done.  You'll have to restart the webservice before that takes effect.
[21:38:45] <gifti>	 ok, thank you!
[22:15:38] <legoktm>	 !log extdist manually deleted /srv/src/extensions/mw-conf.json to force it to pick up the REL1_24 branches on the next run
[23:19:52] <Polsaker>	 I can't connect to tools-login: ssh: connect to host tools-login.wmflabs.org port 22: Connection timed out