[00:00:32] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2935268 (10tom29739) The current version that is on Tool Labs, v0.10.25, was made end of life nearly 3 months ago. This means that it's a security risk.
[00:07:06] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Labs: update image builders to use new PAM scheme - https://phabricator.wikimedia.org/T120710#2935287 (10Andrew) 05Open>03Resolved a:03Andrew
[00:27:15] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2932218 (10bd808) https://packages.debian.org/jessie/nodejs -- `0.10.29~dfsg-2`. That would be the "latest" version for our Kubernetes containers.
[00:33:44] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2935356 (10Tarrow) According to https://github.com/nodejs/LTS 0.10.x and 0.12.x are now both EOL. The former at 2016-10-31 and the later at 2016-12-31.
[00:35:18] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2935357 (10bd808) It looks like we could also 4.2.4 via https://apt.wikimedia.org/wikimedia/pool/main/n/nodejs/ which I think is the version that is being used for some of the production services.
[00:46:08] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2935374 (10Tarrow) I'd be happy with this if it's preferable to using the ubuntu/debian binaries from nodesource (https://github.com/nodesource/distributions) which seems to be what upstream recommend...
[00:48:13] <shinken-wm>	 PROBLEM - High iowait on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: tools.tools-webgrid-generic-1403.cpu.total.iowait (>11.11%)
[00:58:14] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-generic-1403 is OK: OK: All targets OK
[01:17:44] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2935439 (10bd808) >>! In T155063#2935356, @Tarrow wrote: > According to https://github.com/nodejs/LTS 0.10.x and 0.12.x are now both EOL. The former at 2016-10-31 and the later at 2016-12-31.  I would...
[01:25:02] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[01:54:57] <wikibugs>	 (03CR) 10Lokal Profil: [C: 04-1] "changing to -1 due to the percentage comment" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/331407 (owner: 10Jean-Frédéric)
[02:05:01] <shinken-wm>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[04:58:09] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:01:12] <shinken-wm>	 PROBLEM - High iowait on tools-grid-master is CRITICAL: CRITICAL: tools.tools-grid-master.cpu.total.iowait (>33.33%)
[05:02:28] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[05:02:46] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[05:03:00] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.046 second response time
[05:03:58] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[05:04:22] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[05:05:10] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[05:05:20] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[05:05:48] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[05:09:33] * andrewbogott is looking at ^
[05:11:13] <shinken-wm>	 RECOVERY - High iowait on tools-grid-master is OK: OK: All targets OK
[05:17:41] <Krenair>	 andrewbogott, might be related to the stuff going on in -operations?
[05:19:05] <andrewbogott>	 Krenair: yeah, the puppet runs themselves seem to work
[05:19:36] <madhuvishy>	 yeah everything recovered - labstore1004 went down for a bit
[05:19:42] <madhuvishy>	 rather
[05:20:10] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:20:52] <madhuvishy>	 the theory is that a network switch connected to a row got restarted
[05:21:04] <madhuvishy>	 and icinga complained that it was down
[05:30:19] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:39:21] <shinken-wm>	 RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:40:49] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:42:29] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:42:47] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:43:59] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:37:32] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[06:59:09] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:05:36] <wikibugs>	 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2932218 (10Legoktm) >>! In T155063#2935439, @bd808 wrote: >>>! In T155063#2935356, @Tarrow wrote: >> According to https://github.com/nodejs/LTS 0.10.x and 0.12.x are now both EOL. The former at 2016-1...
[07:17:33] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:34:07] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:11:51] <Urbanecm>	 Hi, how can I fix this error connecting using mosh http://pastebin.com/XvRauh6p ?
[09:12:28] <zhuyifei1999_>	 !log video depooling encoding02 + pooling encoding01 + restarting frontend, to upgrade youtube_dl
[09:12:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[09:16:58] <wikibugs>	 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review, 07Puppet, 15User-bd808: Make role::labs::mediawiki_vagrant work on Debian Jessie host systems - https://phabricator.wikimedia.org/T154340#2935839 (10akosiaris)
[10:47:21] <wikibugs>	 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: StewardBot not logged into IRC - https://phabricator.wikimedia.org/T149265#2935925 (10MarcoAurelio)
[12:17:30] <wikibugs>	 (03Draft2) 10MarcoAurelio: Fix date. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/331845
[12:17:36] <wikibugs>	 (03Draft1) 10MarcoAurelio: Fix date. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/331845
[12:19:35] <wikibugs>	 (03CR) 10MarcoAurelio: [C: 032] Fix date. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/331845 (owner: 10MarcoAurelio)
[12:19:38] <wikibugs>	 10Wikibugs: Changes to Gerrit IRC channels - https://phabricator.wikimedia.org/T155165#2935974 (10TTO)
[12:20:23] <wikibugs>	 (03Merged) 10jenkins-bot: Fix date. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/331845 (owner: 10MarcoAurelio)
[14:06:27] <wikibugs>	 10Wikibugs: Changes to wikibugs' IRC channel configuration - https://phabricator.wikimedia.org/T155165#2936138 (10TTO)
[14:14:32] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Malyn was created, changed by Malyn link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Malyn edit summary: Created page with "{{Tools Access Request |Justification=Creating Tools |Completed=false |User Name=Malyn }}"
[14:16:26] <wikibugs>	 06Labs, 10Tool-Labs, 15User-Urbanecm: [request for help] Connection to the toollabs gives locale error - https://phabricator.wikimedia.org/T155172#2936159 (10Urbanecm)
[14:29:10] <mschwarzer>	 hi all. is anybody using hadoop/yarn on labs? i have problems settings up a ssh connection between nodes.
[14:43:44] <wikibugs>	 06Labs, 10Tool-Labs, 15User-Urbanecm: [request for help] Connection to the toollabs gives locale error - https://phabricator.wikimedia.org/T155172#2936257 (10Urbanecm) 05Open>03Invalid Was solved by myself, sorry for creating the ticket! I commented out SendEnv LANG LC_* in my /etc/ssh/ssh_config.
[15:31:58] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:39:37] <wikibugs>	 06Labs, 10Striker, 06Security-Team, 10Wikimedia-Site-requests, and 3 others: Add user group to wikitech granting the oathauth-api-all right - https://phabricator.wikimedia.org/T153487#2936410 (10bd808) a:03bd808
[17:12:02] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-worker-1003 is OK: OK: tools.tools-worker-1003.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) tools.tools-worker-1003.diskspace._public_dumps.byte_percentfree (No valid datapoints found)
[17:15:16] <shinken-wm>	 RECOVERY - Puppet staleness on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [3600.0]
[17:16:59] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:25:38] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Malyn was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1311449 edit summary: 
[18:51:15] <wikibugs>	 (03Draft1) 10Paladox: Add mediawiki/extensions/Babel to #wikimedia-dev for gerrit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/331899 (https://phabricator.wikimedia.org/T155165)
[18:51:17] <wikibugs>	 (03Draft2) 10Paladox: Add mediawiki/extensions/Babel to #wikimedia-dev for gerrit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/331899 (https://phabricator.wikimedia.org/T155165)
[20:57:30] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2936808 (10Tobias1984) 05Open>03Resolved a:03Tobias1984 Thank you all for you help! I contacted @akosiaris
[21:16:54] <wikibugs>	 06Labs, 10DBA, 10Wikidata, 07Performance, and 3 others: Create a new project in labs for testing RedisLock in Wikidata - https://phabricator.wikimedia.org/T155042#2936832 (10Ladsgroup) @chasemp It would be great if you bump the quota specially in number of instances. Since we cleaned it up, it's possible t...
[21:19:46] <Amir1>	 !log wikidata-dev apply labs::mediawiki_vagrant roles to redis-dispatching-client and redis-dispatching-repo (T155190)
[21:19:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[21:19:50] <stashbot>	 T155190: Build an environment to test dispatching - https://phabricator.wikimedia.org/T155190
[22:24:22] <Amir1>	 andrewbogott: Hey, I'm trying to use vagrant in two instances in labs. I followed the manual but I get this:  https://usercontent.irccloud-cdn.com/file/hqZ1TXa5/
[22:24:54] <Amir1>	 nor google nor wikitech had an explanation for this 
[22:25:10] <Amir1>	 Should I file a bug or it's something obvisou I'm missing 
[22:25:24] <andrewbogott>	 Amir1: You're using the labs-vagrant classes, or just mediawiki vagrant?
[22:25:35] <Amir1>	 mediawiki_vagrant 
[22:25:49] <andrewbogott>	 ok — I don't have much idea if that would work on a labs instance.  I'd expect not.
[22:26:31] <andrewbogott>	 oh, wait, you're using the mediawiki_vagrant puppet class, set on horizon?
[22:26:52] <andrewbogott>	 That I /would/ expect to work, but I've never tried it and never touched the code
[22:27:41] <Amir1>	 andrewbogott: https://horizon.wikimedia.org/project/instances/140424db-0eb3-4fc7-b7d5-d7a1424b2b1f/
[22:27:52] <Amir1>	 it's labs::mediawiki_vagrant 
[22:27:57] <andrewbogott>	 ok
[22:28:03] <andrewbogott>	 So, that sounds like the right thing to do
[22:28:08] <andrewbogott>	 but I don't have any idea if/how it works
[22:28:22] <andrewbogott>	 Might be worth mailing labs-l, or hoping that someone else shows up here who has done it
[22:28:26] <Amir1>	 I use it in mediawiki-ores instance too and it worked just fine (but I didn't set it up)
[22:28:43] <andrewbogott>	 iirc it only works on Trusty
[22:28:51] <Amir1>	 yes, it's trusty
[22:34:45] <Amir1>	 madhuvishy: I think switching to NFS is causing this issue ^
[22:34:51] <bd808>	 Amir1: was that error on a brand new instance? Just yesterday we rolled out some puppet changes and a new Vagrant binary. 
[22:35:08] <Amir1>	 bd808: Yup, it was brand new
[22:35:47] <bd808>	 It was working in my tests last week, but I haven't done more testing post deploy. There may be some problems I missed in testing
[22:36:35] <bd808>	 Would you file a bug with that screenshot and assign it to me? I'll poke at it sometime today
[22:36:47] <Amir1>	 bd808: on it
[22:36:56] <bd808>	 Thanks much
[22:38:21] <bd808>	 You might want to try 'vagrant halt; vagrant up' just to see if the classic turn it off and on again shakes something loose
[22:39:45] <bd808>	 That error message is pretty much saying that Vagrant and the OS are not agreeing on how to setup the network connection between the OS and the VM
[22:40:51] <wikibugs>	 06Labs, 10Labs-Infrastructure, 15User-Ladsgroup: Vagrant can't provision - https://phabricator.wikimedia.org/T155196#2937071 (10Ladsgroup)
[22:40:57] <Amir1>	 bd808: https://phabricator.wikimedia.org/T155196 
[22:41:12] <Amir1>	 maybe the documentation is outdated 
[22:41:18] <Amir1>	 https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs
[22:42:32] <Amir1>	 bd808: vagrant reload is basically the same, but I gave it a try just now and didn't work
[22:43:14] <wikibugs>	 06Labs, 10Labs-Infrastructure, 15User-Ladsgroup: Vagrant can't provision - https://phabricator.wikimedia.org/T155196#2937085 (10Ladsgroup)
[22:44:06] <bd808>	 It sounds like an issue with something in the host OS Puppet or software config that it sets up
[22:44:33] <bd808>	 There were some new changes there and my testing may have missed something 
[22:44:35] <madhuvishy>	 hello - i saw nfs something but i guess it's something else?
[22:45:39] <madhuvishy>	 labs nfs hasn't changed yet btw - only tools has
[22:45:51] <Amir1>	 madhuvishy: oh, sorry
[22:46:05] <madhuvishy>	 Amir1: np :)
[22:46:41] <Amir1>	 bd808: hmm, so it's a problem in the puppet role I guess 
[22:49:35] <wikibugs>	 06Labs, 10Labs-Infrastructure, 15User-Ladsgroup: Vagrant can't provision - https://phabricator.wikimedia.org/T155196#2937094 (10Ladsgroup)
[22:50:30] <bd808>	 Amir1: yeah, likely some mismatch between the new Vagrant version and the config that is supposed to make it go
[22:54:15] <wikibugs>	 06Labs, 10Labs-Infrastructure, 15User-Ladsgroup: Vagrant can't provision - https://phabricator.wikimedia.org/T155196#2937102 (10daniel) p:05Normal>03High Bumping to high. This is blocking the resolution of a long standing db performance issue.
[23:00:51] <wikibugs>	 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup: Vagrant can't provision - https://phabricator.wikimedia.org/T155196#2937133 (10bd808) This is likely related to changes that landed very recently for {T155112} and {T154340}.
[23:01:26] <wikibugs>	 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2937151 (10bd808)
[23:04:36] <wikibugs>	 06Labs, 10MediaWiki-Vagrant, 15User-Ladsgroup: Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant - https://phabricator.wikimedia.org/T155196#2937161 (10Ladsgroup) This is last lines of the error when debug mode is enabled (using VAGRANT_LOG=DEBUG ) ```  INFO environment: Released p...