[00:53:32] ebernhardson: opendj. https://wikitech.wikimedia.org/wiki/Labs_infrastructure [01:04:55] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048724 (10Andrew) First, to clarify: the labs-private repo, although poorly named, is just as (un) private as we want it. It will most likely be replaced by something using Hiera eventu... [01:05:45] 3Wikimedia-Labs-wikitech-interface, operations: wikitech instances list is blank - https://phabricator.wikimedia.org/T89808#1048726 (10Andrew) I just re-ran the smw rebuild, so that might have fixed half of this. [01:21:11] 3Tool-Labs: Trusty doesn't have "at" installed by default - https://phabricator.wikimedia.org/T72324#1048768 (10scfc) p:5Triage>3Normal [01:30:40] !ping [01:30:40] !pong [02:26:00] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048901 (10Dzahn) @kartik usually how it works is that you ask ops to add the private thing into the (really private) ops/private repo (how we do it with passwords as well), and then you... [03:10:33] Yuvi|Vacation: Did anything happen to redis last Friday at around 1745 GMT? [03:19:46] a930913: yes, it ran out of space and died so we created a new instance that's bigger [03:21:36] legoktm: Ah, it silently failed and left stale connections open all over my stuff :p [03:21:46] :( [04:14:00] 3Labs: Two instances with same name - https://phabricator.wikimedia.org/T89931#1048981 (10scfc) 3NEW [06:57:50] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:22:45] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:37] Anyone having connection problems to Labs? I can't connect via mosh at all, and SSH freezes after the connection establishes. [09:57:16] Never mind, there must be something wrong with my network connectivity. :) [11:08:41] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<55.56%) [11:43:44] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [12:03:59] Getting ruby errors when trying to run labs-vagrant on a new instance https://phabricator.wikimedia.org/P311 [12:07:21] Any idea what might be wrong? [12:14:04] 3Engineering-Community, Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1049581 (10Aklapper) [13:49:08] RECOVERY - Host tools-webproxy-jessie is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [14:06:56] 3Engineering-Community, Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1049693 (10Aklapper) [14:15:57] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:45:57] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [14:57:30] Hey guys, I don't know if there is something wrong or if it's me, but I don't see any instance for project analytics :/ [15:02:13] Logout / login solved it [15:02:20] Sorry for disturbance :) [15:28:12] 3Engineering-Community, Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1049811 (10Ricordisamoa) >>! In T87730#1000846, @coren wrote: > The "real" solution remains to hound maintainers to make certain they are not alone with access to a t... [15:38:32] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1049875 (10BBlack) I suspect the issue here is that there's a 3rd class of data privacy in play: API keys and such that aren't as private as our production-private stuff, but which we'd r... [17:01:34] PROBLEM - Host tools-webproxy-jessie is DOWN: CRITICAL - Host Unreachable (10.68.17.147) [17:40:42] having some trouble with deployment-prep instances and sudo access. Logged into deployment-salt.eqiad.wmflabs with my shell access user through bastion-eqiad.wmflabs.org. Running `sudo which sudo` prompts me for a password. Entering my ldap pass nets: "thcipriani is not allowed to run sudo on deployment-salt. This incident will be reported." [17:46:13] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1050200 (10coren) >>! In T89642#1049875, @BBlack wrote: > API keys and such that aren't as private as our production-private stuff, but which we'd rather not blast out to the entire plane... [17:49:45] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1050214 (10KartikMistry) I would like to point that this also applies to Beta Cluster. [18:00:00] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1050241 (10Krenair) I'm assuming you have a particular secret in mind that you want to put in deployment-prep somewhere. What instances would need to be able to access that exactly? Some... [19:22:23] 3Tool-Labs: Java jobs stop working - https://phabricator.wikimedia.org/T88799#1050724 (10dnaber) `qacct -j fr-feedcheck | grep maxvmem | grep -v "0.000"` gives this: ``` maxvmem 1.440G maxvmem 1.706G maxvmem 1.440G maxvmem 1.441G maxvmem 1.258G maxvmem 1.453G maxvmem 1.721G ma... [19:44:09] RECOVERY - Host tools-webproxy-jessie is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [20:19:29] 3Labs: Labs web proxy should be load-balanced and tolerate the failure of virt host - https://phabricator.wikimedia.org/T89995#1051019 (10Andrew) 3NEW [20:21:56] 3Labs: Labs web proxy should be load-balanced and tolerate the failure of virt host - https://phabricator.wikimedia.org/T89995#1051027 (10coren) It seems to me unlikely that the proxy can be made HA without a redesign, but having a warm standby ready to take over at the flip of a switch would probably be a suffi... [20:22:42] 3Pywikibot-compat-to-core, Tool-Labs, pywikibot-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#1051029 (10jayvdb) Now that T76286 is merged, another dependency for py2.6 and 2.7 is https://pypi.python.org/pypi/ipaddress [20:28:33] 3Labs, ops-eqiad, operations: virt1002 broken disk? - https://phabricator.wikimedia.org/T88923#1051044 (10Cmjohnson) a:3Cmjohnson [20:33:31] petan: re: the hackathon… if you have any interest/good ideas about https://phabricator.wikimedia.org/T89995 I’d be happy to add you as a coworker on that. [20:54:36] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<12.50%) [20:59:41] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [21:15:30] 3Labs: Labs web proxy should be load-balanced and tolerate the failure of virt host - https://phabricator.wikimedia.org/T89995#1051234 (10scfc) I think Labs is too intertwined that having the Labs/Tools proxy back up sooner than the rest is very useful. Fixing this hardware failure for the most part in less tha... [21:16:26] Hi hello , i would like to have the shell access . so could some one add me to the shell [21:16:37] and my account is kartheek3011 [21:16:47] kartheek: one moment... [21:17:33] kartheek: looks like it was done already :) [21:17:52] thank you [21:17:55] andrew [22:17:30] 3Tool-Labs: Missing or wrong information in meta_p.wiki table - https://phabricator.wikimedia.org/T56962#1051546 (10jayvdb) Is T69476 part of the scope of this task? [22:26:06] 3Labs: Labs web proxy should be load-balanced and tolerate the failure of virt host - https://phabricator.wikimedia.org/T89995#1051571 (10Qgil) @Andrew, you are proposing to work on this task during the Wikimedia Hackathon in Lyon. Please consider associating this task with #Wikimedia-Hackathon-2015. [22:29:38] 3Wikimedia-Hackathon-2015, Labs: Labs web proxy should be load-balanced and tolerate the failure of virt host - https://phabricator.wikimedia.org/T89995#1051585 (10Andrew) [22:46:29] 3Labs, ops-eqiad, operations: virt1002 broken disk? - https://phabricator.wikimedia.org/T88923#1051614 (10Jgreen) As has been discussed elsewhere, check-raid.py only checks the first RAID variant it finds, in this case it's reporting mdadm status. However even if it were making it throught to the mpt check, it...