[00:03:36] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2255546 (10IKhitron)
[01:36:18] <TParis>	 Hey, if MediaWiki throws a captcha from the API, is there a way to get it to stop doing that?
[01:36:23] <TParis>	 Does it need to be confirmed?
[02:37:15] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:53:13] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:14] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824240 bytes in 9.097 second response time
[05:01:43] <shinken-wm>	 RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:13:30] <wikibugs>	 06Labs, 10Labs-Infrastructure: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2255706 (10Kelson)
[05:13:46] <wikibugs>	 06Labs, 10Labs-Infrastructure: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2255719 (10Kelson)
[06:55:49] <grrrit-wm>	 (03PS2) 10Yuvipanda: robots.txt: disallow /persondata/ [labs/toollabs] - 10https://gerrit.wikimedia.org/r/286315 (https://phabricator.wikimedia.org/T133697) (owner: 10Merlijn van Deen)
[06:56:01] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] robots.txt: disallow /persondata/ [labs/toollabs] - 10https://gerrit.wikimedia.org/r/286315 (https://phabricator.wikimedia.org/T133697) (owner: 10Merlijn van Deen)
[07:06:36] <valhallasw`cloud>	 YuviPanda: https://tools.wmflabs.org/robots.txt wheeee
[07:06:53] <YuviPanda>	 \o/
[07:07:14] <YuviPanda>	 dump-stats.eqiad.wmflabs is hammering NFS and I can't get on it
[07:07:22] <YuviPanda>	 even with root?!
[07:07:36] <valhallasw`cloud>	 O_o
[07:07:49] <YuviPanda>	 because I have a typo. nevermind.
[07:08:05] <valhallasw`cloud>	 we should really log sshd to the console log :-p
[07:13:10] <wikibugs>	 06Labs: dump-stats instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2255844 (10yuvipanda)
[08:34:52] <shinken-wm>	 PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228)
[08:51:09] <wikibugs>	 10Labs-Other-Projects, 13Patch-For-Review: Mount dumps to the mlp instance of the math project - https://phabricator.wikimedia.org/T134026#2256073 (10yuvipanda) 05Open>03Resolved Next puppet run should make this mount show up.  Thanks for filing the bug!
[09:12:02] <tom29739>	 How can I specify a release with qlogin? I keep getting a precise instance, when I need a trusty one to see why my tool isn't working on the grid.
[09:13:08] <YuviPanda>	 I think qlogin is present, but as tools admins I would say we do not have the bandwidth to support its use, and you're on your own. sorry!
[09:13:13] <valhallasw`cloud>	 tom29739: -l release=trusty ?
[09:14:17] <valhallasw`cloud>	 you can also just ssh to the relevant host to investiage
[09:14:34] <YuviPanda>	 ^
[09:14:37] <tom29739>	 That would make sense.
[09:14:50] <YuviPanda>	 that's a supported option :)
[09:39:12] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Set-up unit-tests for the monuments API in PHP [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286402 
[09:43:34] <grrrit-wm>	 (03PS2) 10Jean-Frédéric: Set-up unit-tests for the monuments API in PHP [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286402 
[09:54:29] <gehel>	 !log deployment-prep restart elasticsearch cluster to ensure multicast configuration is disabled (T110236)
[09:54:29] <stashbot>	 Please !log in #wikimedia-releng for beta cluster SAL
[09:54:29] <stashbot>	 T110236: Use unicast instead of multicast for node communication - https://phabricator.wikimedia.org/T110236
[09:54:35] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[10:07:49] <wikibugs>	 06Labs: dump-stats instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2256228 (10Hydriz) Thanks for the notice @yuvipanda. I would avoid using NFS if I could since the files are only used temporarily. The instance storage is ideal, but it is too small for our use-case. Is it possible for us t...
[10:26:41] <wikibugs>	 06Labs: dump-stats instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2255844 (10yuvipanda) From my understanding, the current answer is 'no' :(   Perhaps we can modify the quotas, which would give you the storage you want even if at reduced speeds. I am not fully aware of how this is setup,...
[10:28:37] <wikibugs>	 06Labs, 10Horizon, 13Patch-For-Review: Create puppet backend with REST api for labs instance configuration - https://phabricator.wikimedia.org/T133412#2256245 (10yuvipanda) a:05yuvipanda>03Andrew Re-assigning to andrew to figure out next steps, since I've finished the code now :D
[11:46:03] <grrrit-wm>	 (03CR) 10Lokal Profil: [C: 032] "Thanks!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286402 (owner: 10Jean-Frédéric)
[11:46:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set-up unit-tests for the monuments API in PHP [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286402 (owner: 10Jean-Frédéric)
[12:24:52] <wikibugs>	 06Labs: Put a firewall on labtestcontrol2001 - https://phabricator.wikimedia.org/T132598#2256378 (10MoritzMuehlenhoff) p:05Triage>03Normal
[12:30:47] <grrrit-wm>	 (03CR) 10Lokal Profil: [C: 04-1] "One minor comment, otherwise this is good to go" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 (owner: 10Jean-Frédéric)
[13:05:14] <wikibugs>	 06Labs: dumps-stats.dumps.eqiad.wmflabs instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2256495 (10yuvipanda)
[13:07:55] <grrrit-wm>	 (03PS5) 10Jean-Frédéric: Add unit tests to test_update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 
[13:08:23] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Aaaand done :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 (owner: 10Jean-Frédéric)
[13:09:25] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Inline comment." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[13:23:19] <wikibugs>	 06Labs: dumps-stats.dumps.eqiad.wmflabs instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2256568 (10yuvipanda) @Hydriz can you also tell me what this 1T is being used for, so I can understand the usecase better?
[13:34:28] <tom29739>	 I'm getting an error when I submit my command to jstart. It says: 'Fatal Python error: Couldn't create autoTLSkey mapping /var/spool/gridengine/execd/tools-exec-1410/job_scripts/5895593: line 4:  5531 Aborted                 (core dumped)'. What does that mean? It works when I run it from the bastion.
[13:36:25] <gifti>	 does it have enough memory?
[13:36:52] <YuviPanda>	 tom29739: googling for the error message indicates it could be a memory issue (http://stackoverflow.com/questions/18638518/fatal-python-error-couldnt-create-autotlskey-mapping-with-c-binding). Try setting a higher memory limit with the '-mem' option to jsub? 2G maybe? 1G?
[13:40:44] <tom29739>	 It worked. Thanks YuviPanda 
[13:40:56] <YuviPanda>	 np tom29739.
[13:42:39] <wikibugs>	 06Labs: dumps-stats.dumps.eqiad.wmflabs instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2256613 (10Hydriz) Generally it is used to store datasets that are ready to be uploaded to the Internet Archive. This space is used to store the data first as we download them before we upload everythin...
[14:01:57] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256680 (10Reedy) I suspect column number isn't the most user friendly, specifying the actual could be more useful  So from https://quarry.wmflabs.org/run/84084/output/0/json-li...
[14:03:35] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2255546 (10yuvipanda) You can also get the latest output for any query with the following URL format:  `/query/<int:query_id>/result/latest/<string:resultset_id>/<string:format>`
[14:10:38] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256705 (10IKhitron) Thank you that you are interested. About column name - I thought about this, but sometimes it's not a good choice, because column name can be something as...
[14:23:06] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256735 (10Reedy) Teaching a little SQL seems easier...  ``` concat((case page_namespace when 0 then "" when 4 then "Project" when 14 then ":Category" when 118 then "Draft" when...
[14:25:03] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256736 (10IKhitron) I knew about AS, of course. I just wanted make it possible usage of existing queries without any need to change them.
[14:55:33] <grrrit-wm>	 (03PS1) 10Addshore: Add purtle to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/286451 
[14:56:58] <grrrit-wm>	 (03CR) 10Lokal Profil: [C: 032] "Nice!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 (owner: 10Jean-Frédéric)
[15:00:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add unit tests to test_update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 (owner: 10Jean-Frédéric)
[15:21:53] <wikibugs>	 10Tool-Labs-tools-stewardbots, 06translatewiki.net, 05Continuous-Integration-Scaling: labs/tools/stewardbots  and translatewiki repos do not pass 'composer validate' - https://phabricator.wikimedia.org/T134182#2256913 (10hashar)
[15:21:53] <wikibugs>	 10Tool-Labs-tools-stewardbots, 06translatewiki.net, 05Continuous-Integration-Scaling: labs/tools/stewardbots  and translatewiki repos do not pass 'composer validate' - https://phabricator.wikimedia.org/T134182#2256926 (10hashar)
[15:24:09] <wikibugs>	 10Tool-Labs-tools-stewardbots, 06translatewiki.net, 05Continuous-Integration-Scaling: labs/tools/stewardbots  and translatewiki repos do not pass 'composer validate' - https://phabricator.wikimedia.org/T134182#2256936 (10hashar) 05Open>03Invalid Made a mistake, they are not packages :-)
[15:27:38] <YuviPanda>	 valhallasw`cloud: ^ do you want to try pushing / building grrrit-wm1 again, with addshore's patch?
[15:28:13] <valhallasw`cloud>	 YuviPanda: sorry, busy
[15:28:30] <YuviPanda>	 valhallasw`cloud: np, I'll do it
[15:28:44] <YuviPanda>	 addshore: do you have any more patches coming or just this?
[15:34:12] <addshore>	 just that :)
[15:35:53] <YuviPanda>	 addshore: kk
[15:42:37] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Write a k8s admission controller to enforce that all containers running come from our private repository - https://phabricator.wikimedia.org/T133515#2256984 (10yuvipanda) a:03yuvipanda
[15:45:52] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Use a 'pause' container from our private repo, not gcr.io - https://phabricator.wikimedia.org/T133873#2257004 (10yuvipanda) a:03yuvipanda
[15:46:27] <grrrit-wm1>	 (03CR) 10Yuvipanda: [C: 032] Add purtle to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/286451 (owner: 10Addshore)
[15:55:43] <YuviPanda>	 addshore: deploying now
[16:05:10] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Downgrade requirement of phpunit from 5.3.x to 4.8.x [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 
[16:06:07] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Downgrade requirement of phpunit from 5.3.x to 4.8.x [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric)
[16:07:29] <YuviPanda>	 addshore: let me know if it's good
[16:09:05] <wikibugs>	 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2257063 (10IKhitron) To explain you, what did I mean about encoding. When I run Quarry, I never can just use the results. I open the CSV download in Notepad++, convert it to UTF...
[16:14:52] <addshore>	 YuviPanda: it didnt explode ;)
[16:15:07] <YuviPanda>	 addshore: \o/
[16:24:38] <wikibugs>	 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2257101 (10EBernhardson) This project does replace nobelium. Nobelium will be decommissioned and returned to the pool.   I think the differenc...
[16:50:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2257220 (10hashar) p:05Triage>03Normal There is no urgency for now. Nodepool has 2 Trusty for base pool and will spawn...
[16:56:34] <wikibugs>	 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2158516 (10yuvipanda) So my understanding of the commitment from the labs team is:  1. We poke some holes in the labs-instance/labs-support fi...
[17:02:13] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2257253 (10yuvipanda)
[17:10:19] <wikibugs>	 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2257267 (10EBernhardson) Sounds accurate to me
[18:00:44] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2257381 (10bd808)
[20:27:49] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2257668 (10bd808)
[20:27:51] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 06WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2257669 (10bd808)
[20:28:05] <wikibugs>	 06Labs, 06WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#2257670 (10bd808)
[20:28:08] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 06WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#997462 (10bd808)
[21:17:46] <grrrit-wm>	 (03PS3) 10Lokal Profil: [NOT TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) 
[21:45:05] <wikibugs>	 06Labs, 10Tool-Labs: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2257889 (10russblau)
[22:08:01] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "check" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[22:36:43] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] [NOT TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[22:44:05] <grrrit-wm>	 (03PS4) 10Lokal Profil: [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) 
[22:45:28] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[23:12:43] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2258192 (10bd808) >>! In T134188#2257751, @mmodell wrote: > Anyway my original offer still stands - it would...