[00:35:15] <wikibugs>	 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2790474 (10Kotz) @valhallasw thank you for commenting.   ssh -T tools-exec-1403 /usr/bin/mono BotHasava.exe works fine. I didn't find any env var justifying this problem. Certainly it's not a problem of tools...
[01:12:17] <sDrewth>	 Can someone please point me to the process to get a tool reallocated ... dead  https://tools.wmflabs.org/pirsquared/iw.php
[01:18:08] <bd808>	 sDrewth: It looks like this tool is running on a PHP version that is incompatible with the codebase. So the code needs to be fixed or the maintainer needs to pin the webservice to run on the older precise exec nodes
[01:19:25] <bd808>	 sDrewth: the maintainer is https://wikitech.wikimedia.org/wiki/User:PiRSquared17
[01:20:06] <sDrewth>	 yes Bryan, and I haven't seen him for ages. I will go and beat heavily on his user talk page
[01:21:24] <bd808>	 one awesome thing is that the code has a license
[01:22:16] <sDrewth>	 did I read that precise is disappearing soon?
[01:23:22] <bd808>	 March is our target for removing it entirely -- https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation
[01:23:43] <sDrewth>	 ok, he should update the code rather than run precise IMNSHO
[01:24:21] <bd808>	 the error is a call_user_func_array() that is passing a variable by reference. I can write up a bug in phab
[01:24:44] <sDrewth>	 I have whined at https://meta.wikimedia.org/wiki/User_talk:PiRSquared17#Fix_for_https:.2F.2Ftools.wmflabs.org.2Fpirsquared.2Fiw.php
[01:25:49] <sDrewth>	 <- just a tool user, not a coder
[01:26:15] <bd808>	 *nod*
[01:28:30] <sDrewth>	 a conceptual skill which bypassed me, waaaaay back when computer studies was a rare course at high school
[01:31:28] <bd808>	 ugh. I think fixing is going to be a bit of work. The function that is being called was removed in PHP 5.4 too.
[01:32:02] <bd808>	 ah. just renamed and the alias removed
[01:45:58] <wikibugs>	 10Tool-Labs-tools-Other: https://tools.wmflabs.org/pirsquared/iw.php crashes with fatal error - https://phabricator.wikimedia.org/T150592#2790492 (10bd808)
[01:46:17] <bd808>	 sDrewth: ^ there's a bug report
[01:46:45] <sDrewth>	 thx
[01:48:52] <sDrewth>	 suppled to user
[01:49:14] <sDrewth>	 oh d'oh, you added them <facepalm>
[01:50:23] <sDrewth>	 we still have issues that these widely functional tools are tucked away in back pockets :-/
[01:51:05] <sDrewth>	 we are still stuck in the toolserver approach of one user accounts and dying tools
[01:51:21] <sDrewth>	 </grump>
[02:39:51] <bd808>	 sDrewth: you are preaching to the wrong guy :) See https://wikitech.wikimedia.org/wiki/User:BryanDavis/Developing_community_norms_for_critical_bots_and_tools for some ideas on the topic from me.
[02:55:10] <sDrewth>	 not preaching bd808, just whinging/reflecting/headthunking
[02:55:41] <bd808>	 sDrewth: I fixed the bug. Give the tool a shot and see if it works now
[02:56:53] <sDrewth>	 oooooo, <click>
[02:58:32] <wikibugs>	 10Tool-Labs-tools-Other: https://tools.wmflabs.org/pirsquared/iw.php crashes with fatal error - https://phabricator.wikimedia.org/T150592#2790557 (10bd808) I made this change to /data/project/pirsquared/public_html/iw.php: ``` lang=php, name=iw.php         /*         call_user_func_array(             array(...
[03:11:07] <sDrewth>	 works beautifully searching for JIRA, though I should have known better when searching for TOOLS (ouch!)
[03:11:33] <sDrewth>	 huge tableeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
[03:18:27] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 1.21 ms
[03:27:04] <sDrewth>	 bd808: how difficult or problematic would be to set up a incoming dead link page for Tools.  We have many many old interwiki_map links that utilise Tools: (aka Toolserver) and I am wondering whether we are better to just have those dead, or to find a way to manage them at tools.wmflabs OR at something like toolserver.wmflabs.org?*  which is just an information page telling them what happened
[03:28:14] <bd808>	 I think we had a redirect server for the old URLs for a couple of years but finally shut it down
[03:29:02] <sDrewth>	 I am not wanting to individually redirect the links, but collectively
[03:29:49] <sDrewth>	 if we could redo the interwiki_map to point to toolserver,wmflabs,org, and that can capture everything thwon at it to one single page of information (or something equally generic)
[03:30:53] <bd808>	 ah. gotcha. I wonder if we could just send them all to https://meta.wikimedia.org/wiki/Toolserver somehow
[03:32:28] <sDrewth>	 how about a compromise. We point to toolserver.wmflabs.org to handle the rewrites and that points to meta page?
[03:32:48] <sDrewth>	 rather than have to handle all the rewrite rules at meta
[03:33:09] <sDrewth>	 all those tilda ~  addresses would be a PITA
[03:33:12] <sDrewth>	 tilde
[03:33:52] <sDrewth>	 hmm, would you prefer that I do a phabricator ticket?
[03:33:57] <bd808>	 well an interwiki link doesn't need to point to a top level domain. It could point to https://meta.wikimedia.org/wiki/Toolserver?tool=$1 I think
[03:34:14] <bd808>	 yes this is a phab ticket worthy idea for sure
[03:35:37] <bd808>	 If someone wanted to get really fancy they could actually make a tool to be the target of the interwiki link and then try to suggest modern tools that do the things the old links do.
[03:36:08] <bd808>	 although really people should be tracking down the links and replacing them on-wiki
[03:37:15] <sDrewth>	 I totally agree, and that is why we usually do with that tool, however many many are on user pages, and relity is that it isn't going to happen
[03:37:40] <sDrewth>	 why = what
[03:42:02] <sDrewth>	 we apparently still have the links in templates, and it is hard enough to just find those!
[03:43:36] <sDrewth>	 woohoo, just found one
[04:19:40] <wikibugs>	 06Labs, 10MediaWiki-Interwiki: All interwiki mapped TOOLS: and JIRA: urls to redirect single url - https://phabricator.wikimedia.org/T150598#2790611 (10Billinghurst)
[05:17:56] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170)
[05:23:32] <wikibugs>	 06Labs, 10MediaWiki-Interwiki: All interwiki mapped TOOLS: and JIRA: urls to redirect single url - https://phabricator.wikimedia.org/T150598#2790627 (10Billinghurst)
[05:24:48] <wikibugs>	 06Labs, 10MediaWiki-Interwiki: All interwiki mapped TOOLS: and JIRA: urls to redirect single url - https://phabricator.wikimedia.org/T150598#2790611 (10Billinghurst) Hmm, there are some redirects that still work, so it is not a complete story. It looks very messy.
[06:21:03] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[06:47:02] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.013 second response time
[06:57:01] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.035 second response time
[07:01:00] <shinken-wm>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:32:53] <wikibugs>	 06Labs, 10Tool-Labs: Perl module problems on 14## exec nodes - https://phabricator.wikimedia.org/T150120#2790639 (10Beetstra) (I temporary killed the bot (tools.xlinkbot) that is now non-functional, I expect others to become problematic in time).  Can the 14## exec nodes please be made 'the same' as the 12## e...
[11:28:00] <wikibugs>	 06Labs, 10Tool-Labs: Perl module problems on 14## exec nodes - https://phabricator.wikimedia.org/T150120#2774533 (10valhallasw) >>! In T150120#2790639, @Beetstra wrote: > Can the 14XX exec nodes please be made 'the same' as the 12XX exec nodes in terms of installed software.  This is breaking stuff.  They are,...
[11:30:28] <wikibugs>	 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2790777 (10valhallasw) >>! In T150099#2790474, @Kotz wrote: > Is there a way I can get the same env as jsub on the login server or in a more or less interactive shell?   You can try `qlogin`, but I'm not sure...
[11:48:15] <grrrit-wm>	 (03PS1) 10Paladox: Add fsf/member/ to list of whitelisted cloaks [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) 
[11:50:45] <grrrit-wm>	 (03CR) 10Peachey88: [C: 04-1] ""fsf/member/" isn't appropriate here, "@fsf/member/marktraceur" should be the entry" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[11:51:45] <grrrit-wm>	 (03CR) 10Paladox: "> "fsf/member/" isn't appropriate here, "@fsf/member/marktraceur"" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[11:59:06] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms
[12:02:45] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22)
[12:04:39] <shinken-wm>	 RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.95 ms
[12:06:40] <shinken-wm>	 PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218)
[14:12:43] <wikibugs>	 06Labs, 10Tool-Labs: Perl module problems on 14## exec nodes - https://phabricator.wikimedia.org/T150120#2790910 (10Beetstra) Well, obviously, they are not the same as the 12## nodes (see also a bug about sudden php errors on 14xx nodes that were not there on the 12xx nodes, bug T149810).  These issues are als...
[16:12:09] <wikibugs>	 06Labs, 10Tool-Labs: Implement a system to monitor tools on tool-labs - https://phabricator.wikimedia.org/T53434#2791004 (10Aklapper)
[16:14:00] <wikibugs>	 06Labs, 10Tool-Labs, 10community-labs-monitoring: Implement a system to monitor tools on tool-labs - https://phabricator.wikimedia.org/T53434#572676 (10Aklapper)
[16:55:58] <wikibugs>	 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154#2791023 (10madhuvishy)
[19:32:30] <wikibugs>	 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2791110 (10Kotz) qlogin and qrsh work OK too.  I can actually do  ``` qrsh -l release=trusty mono BotHasava.exe ```  and this works quite reliably. Is it OK to use qrsh instead of jsub?
[20:15:17] <wikibugs>	 10Tool-Labs-tools-Other: Geohack should be mobile friendly - https://phabricator.wikimedia.org/T103409#2791151 (10Dispenser) 05Open>03Resolved a:03Dispenser The theme is responsive now.  The following CSS classes `nomobile`, `mobileonly`, and `mobile-linearize` have been added.
[20:15:37] <shinken-wm>	 PROBLEM - Host tools-puppetmaster-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.61)
[21:39:51] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-1] "The regex should be:" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:40:15] <grrrit-wm>	 (03CR) 10Alex Monk: "Actually:" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:41:58] <grrrit-wm>	 (03CR) 10Paladox: "But wont that fail? Since it isen't doing /*/" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:42:39] <grrrit-wm>	 (03CR) 10Alex Monk: "No, your \/* was just matching any number of forward slashes, it wasn't useful." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:43:12] <grrrit-wm>	 (03CR) 10Alex Monk: "You probably meant it to be \/.*$ but you don't need to do that" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:43:31] <grrrit-wm>	 (03PS2) 10Paladox: Add fsf/member/ to list of whitelisted cloaks [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) 
[21:45:19] <grrrit-wm>	 (03CR) 10Paladox: "Thanks, @Krenair done. :)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[21:47:35] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] "I tested this and it works." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[22:17:02] <grrrit-wm>	 (03PS3) 10Paladox: Add fsf/member/marktraceur to cloak whitelist regex [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) 
[22:17:05] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Add fsf/member/marktraceur to cloak whitelist regex [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[22:17:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add fsf/member/marktraceur to cloak whitelist regex [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[22:17:48] <grrrit-wm>	 (03CR) 10Paladox: "Thanks." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321174 (https://phabricator.wikimedia.org/T149689) (owner: 10Paladox)
[22:17:52] <marktraceur>	 Why am I on the whitelist
[22:18:55] <paladox>	 marktraceur becuase you review grrrit-wm changes sometimes.
[22:19:14] <marktraceur>	 I can be on the whitelist for the test instance, maybe?
[22:19:31] <paladox>	 Oh.
[22:32:20] <grrrit-wm>	 (03PS1) 10Alex Monk: Remove whitelist_nicks [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321335 
[22:33:22] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Remove whitelist_nicks [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321335 (owner: 10Alex Monk)
[22:33:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove whitelist_nicks [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321335 (owner: 10Alex Monk)
[22:49:19] <grrrit-wm>	 (03PS9) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 
[22:56:46] <grrrit-wm>	 (03PS1) 10Alex Monk: In command handling section, use 4-space indentation (like the rest of the code) and rename variables [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321336 
[22:56:49] <grrrit-wm>	 (03PS1) 10Alex Monk: Simplify command handling section [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321337 
[22:56:52] <grrrit-wm>	 (03PS1) 10Alex Monk: Move around a load more logging, responses etc. [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321338 
[23:00:22] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] "Should I +2 since this looks good?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321336 (owner: 10Alex Monk)
[23:00:46] <grrrit-wm>	 (03CR) 10Alex Monk: "Sure" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321336 (owner: 10Alex Monk)
[23:01:34] <grrrit-wm>	 (03CR) 10Paladox: [C: 032] "Ok thanks" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321336 (owner: 10Alex Monk)
[23:03:20] <grrrit-wm>	 (03CR) 10Paladox: [C: 032] Simplify command handling section [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321337 (owner: 10Alex Monk)
[23:04:07] <grrrit-wm>	 (03CR) 10Alex Monk: "(testing bot)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321335 (owner: 10Alex Monk)
[23:08:35] <paladox>	 grrrit-wm: nick
[23:08:56] <paladox>	 grrrit-wm: nick
[23:08:59] <grrrit-wm>	 Nick is already grrrit-wm not changing the nick.
[23:09:00] <grrrit-wm>	 Nick is already grrrit-wm not changing the nick.
[23:30:23] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2791429 (10Halfak) > on existing wiki's starting with these as X happens to add the table  We have 8 wikis and we're expan...