[00:09:15] 6Labs, 10Tool-Labs: Setup a supported HTTP Ingress solution for Kubernetes - https://phabricator.wikimedia.org/T129312#2138046 (10yuvipanda) [00:09:17] 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2138045 (10yuvipanda) [00:09:31] 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2101957 (10yuvipanda) T129312 is a nice to have, but not required to meet this goal. [00:35:16] 6Labs, 10Tool-Labs: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2138078 (10yuvipanda) Kubernetes 1.2 is out now, so we can use the opportunity to upgrade + use new setup. [00:38:15] 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2138079 (10yuvipanda) So I think the basic work structure of this goal is essentially two 'streams' that can kind of proceed independently: 1. Solidfiy and improv... [01:05:37] 6Labs, 10Labs-Sprint-100, 10Tool-Labs: Deploy new unified webservice code - https://phabricator.wikimedia.org/T98440#2138083 (10yuvipanda) [03:18:03] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:51] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:35:06] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2138253 (10Shizhao) >>! In T120497#2126355, @Yurik wrote: > @Shizhao , please update your example link above - if someon's user interface is not "zh", they won't... [09:59:21] (03CR) 10Hashar: Continuous Integration Python config for labs/tools/stewardbots (031 comment) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [10:07:12] (03CR) 10Hashar: [C: 04-1] Continuous Integration Python config for labs/tools/stewardbots (035 comments) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [12:45:00] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2138485 (10MarcoAurelio) 5Open>3Resolved a:3Glaisher Worksforme :) Thanks. [13:04:25] 10Labs-Other-Projects: Succesful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2138500 (10AdHuikeshoven) [13:50:59] 10Labs-Other-Projects: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2138544 (10jayvdb) [13:52:16] 10Labs-Other-Projects: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#1962887 (10jayvdb) [14:25:38] 6Labs, 10Labs-Infrastructure: Install python-requests-oauthlib on labs - https://phabricator.wikimedia.org/T130529#2138599 (10Billinghurst) [14:26:35] restarting labsdb1001, it got locked due to metadata locking [14:29:47] 6Labs, 10Labs-Infrastructure: Install python-requests-oauthlib on labs - https://phabricator.wikimedia.org/T130529#2138614 (10Billinghurst) To note that it was also suggested by @anomie that I run as trusty, I had tried that and it still seems to fail Logged in on wikisource:en via OAuth as 10.68.16.38,... [14:33:07] tip of the day: if you want to read the a large table in chunks, do not use LIMIT, use WHERE + an index [14:34:03] LIMIT 10000000, 1 will send just 1 row, but it will, in most cases, have to read 10000000 rows to reach that one [15:37:22] 6Labs, 10Tool-Labs: Korean Locale Installation - https://phabricator.wikimedia.org/T130532#2138850 (10Krenair) Okay, in future please say which system you are using instead of leaving everyone else to guess. [15:49:35] 6Labs, 10Tool-Labs: Korean Locale Installation - https://phabricator.wikimedia.org/T130532#2138879 (10Ykhwong) Thanks, Krenair. From now on I will follow your advice before creating a task. [15:53:00] 6Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2138922 (10AlexMonk-WMF) This has been on labtesthorizon for a little while now. The changes aren't puppetised yet but that's only actively an issue for the config... [16:40:13] 6Labs: Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2139066 (10Andrew) [19:03:55] Luke081515: around? [19:04:08] yeah [19:04:12] Luke081515: I realized today you could try to make mysql accessible from the outside by fiddling with the firewall [19:04:27] if it's running on all interfaces, at least (not sure if that's the case) [19:04:31] valhallasw`cloud: I don [19:04:34] argh [19:04:52] I don't know, if this makes a differences but the mysql is in a virtual machine at this instance [19:05:17] oh, it's inside vagrant... that might make life more complicated [19:06:26] valhallasw`cloud: Otherwise resetup this is not a big problem, mit logic just was: "if it's easy to recover, that it's better to do it this way that resetup" [19:06:32] *nod* [19:06:40] but if it's really complcated I don't want to make you much work [19:17:12] Luke081515: if I understand andrew correctly, it's basically impossible to login -- I suspect he ran commands through salt [19:17:19] which is possible, but painful and slow [19:17:30] and it might not be possible to get into the internal vm [19:18:10] If I understand the task right, the problem is, that /var is chowned away from root. Is it not possible to chown it to root again? This could fix the issue [19:18:59] that's likely to cause a whole load of other problems [19:25:12] valhallasw`cloud: Why? To ask in another way: Before deleting that instance, maybe one possible try to backup the data? [19:25:31] Luke081515: I suppose it might work, yes [19:27:51] I mean, if this will cause other problems at other parts at labs, we should not to that, but if not, give it a try could not be wrong [19:29:46] *nod*. please ask andrewbogott, as noted, I don't have the access to do so [19:30:12] ok [19:36:11] Luke|Dinner: there's really no limit to how messed up that box is at this point [19:36:38] obviously the directories that you chown -R'd had subdirs owned by all different people and services [19:36:50] all that knowledge has been erased for good [19:37:08] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:37:14] I already did a chown -R /var a few days ago but it just means that things are now broken in different ways [19:43:11] 6Labs, 10Tool-Labs: tools-exec-1202 puppet failure: http://security.ubuntu.com/ubuntu/ precise-security/universe git-svn all 1:1.7.9.5-1ubuntu0.3 not found - https://phabricator.wikimedia.org/T130556#2139328 (10valhallasw) [19:45:59] 6Labs, 10Tool-Labs: tools-exec-1202 puppet failure: http://security.ubuntu.com/ubuntu/ precise-security/universe git-svn all 1:1.7.9.5-1ubuntu0.3 not found - https://phabricator.wikimedia.org/T130556#2139343 (10valhallasw) 5Open>3Resolved a:3valhallasw Seems to be resolved after a manual `apt-get update`... [20:30:29] andrewbogott: Ok, thanks [20:37:59] andrewbogott: In theory, I looked up a fresh instance with data like that, there are only a few directorys to change. One of the labs team said, that it could be, that the not running puppet is the problem. Is there a difference, if you set 'chown -R puppet /var/lib/puppet' ? [21:14:47] !log tools.luke081515bot Deploying the 16 commits from today to tools [21:14:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.luke081515bot/SAL, Master [22:02:27] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [22:58:56] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:06] (03PS1) 10Alexandros Kosiaris: Add ores::redis::password [labs/private] - 10https://gerrit.wikimedia.org/r/278837 [23:04:49] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add ores::redis::password [labs/private] - 10https://gerrit.wikimedia.org/r/278837 (owner: 10Alexandros Kosiaris)