[00:11:48] valhallasw`cloud: ugh (re: nfs client) [00:11:54] valhallasw`cloud: I wonder if upgrading kernels will help [00:16:10] YuviPanda: we should try gluster! [00:16:12] ;-) [00:16:29] YuviPanda: also: happy new year! [00:17:37] valhallasw`cloud: happy new year to you too! :D [00:20:20] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6Release-Engineering-Team, and 3 others: rake-jessie jobs stuck due to no ci-jessie-wikimedia slaves being attached to Jenkins - https://phabricator.wikimedia.org/T122731#1912365 (10bd808) Restored ~2016-01-011T23:58 @ori and @legoktm... [00:20:34] valhallasw`cloud: I wonder if I can upgrade kernel on a few hosts and see what happens [00:21:49] I'm trying to imagine how to break https://wikitech.wikimedia.org/wiki/Help:Tool_Labs up into actually readable chunks... [00:22:42] bd808: actually sometimes I prefer a longer page... [00:22:56] so I can Ctrl+F for a keyword and find the relevant part immediately [00:23:25] *nod* a substitute for good site search [00:23:57] that page just seems to be too much for "why and how to use Tool Labs" [00:24:29] bd808: I think we should have more of a read the docs like interface [00:24:46] Mediawiki is just not very good for these situations [00:25:23] But it makes editing easy, which is also important... [00:25:33] You mean organizing large amounts of information...? [00:25:44] I think enwiki might disagree [00:25:47] Manual-type information [00:26:16] bd808: it doesn't do sectioning and stuff properly [00:26:26] the sidebar of exploratory stuff + search on rtd is super useful [00:26:36] Enwiki and physics and math articles makes me say it also doesn't work that well there either ;) [00:28:25] I think we should also step back and rethink the entire 'flow' [00:28:27] and work from there [00:28:52] https://wikitech.wikimedia.org/wiki/User:BryanDavis/NewPortals [00:29:34] bd808: nice! [00:29:43] a link to http://wikitech.wikimedia.org/wiki/Labs_labs_labs somewhere might be nice too [00:29:46] but yeah, +1 [00:30:02] not sure who (you?) suggested a Tool: namespace on wikitech [00:30:05] which I +1 too [00:30:27] Yeah I said it in an email recently [00:30:44] for giving a home to project docs [00:32:00] Coren proposed and then got talked out of a separate wiki in T70818. I kind of hope that some namespaces and search boosts may help with the core problems [00:32:08] * YuviPanda nods [00:32:41] hopefully in the next 6 months we can kill OpenStackManager too [00:34:12] bd808: valhallasw`cloud I've been struggling to figure out which of the 2 next-scenarios in https://etherpad.wikimedia.org/p/tools-kubernetes-scenarios to go for next [00:34:30] one is the backwards compat interface - aka just replace gridengine with k8s [00:34:36] NFS stays, all current commands work, etc [00:34:50] other is a PaaS like Deis or OpenShift (Not gonna write our own nope!) [00:35:06] I think the PaaS is a better idea personally [00:35:09] the problem with the PaaS is that they aren't really ready yet and not for another 6 months at least [00:35:13] bd808: yeah that's my thinking too [00:35:34] because it will give new features to incentivize migration [00:35:38] bd808: but I definitely don't think we should build our own, and IMO neither Deis nor OpenShift are ready yet. Deis isn't even alpha until end of month [00:35:44] rather than "oops we changed it all again" [00:35:56] bd808: I don't think the backwards compatible interface will go away [00:36:07] yeah we should NOT built it in house [00:36:17] moving from NFS dependent setup to a PaaS takes work, and I suspect it'll be at least a few years before all of them move off [00:37:16] the b/c interface also allows us to get rid of GridEngine [00:38:27] so you'd attempt to keep the current "porcelain" (git's word for cli) but change the backend to use some home grown thing on k8s nodes? [00:38:40] yes basically [00:38:44] not that home-grown, mostly. [00:39:09] our whole webservice infrastructure can be replaced with k8s replicationcontrollers + services [00:39:30] only 'homegrown' thing is we'll have a single container that replicates our current enviornment [00:39:44] and mount NFS from that and run code [00:39:54] we'll basically be replacing our worker VMs with containers [00:40:01] except each tool will run in one container [00:40:05] *nod* [00:41:12] would all that be easier as a layer on top of the PaaS system when it exists? [00:41:42] meaning mostly would it be better to spend 3 months helping upstream than building custom stuff that will age and crumble? [00:42:47] bd808: I don't think it'll be easier with PaaS [00:42:53] bd808: since the PaaS will be a lot more opinionated [00:44:15] Oh did I tell you that I got mw-vagrant running using LXC on jessie? [00:44:22] oooh nice [00:44:27] I made it work on my laptop at least [00:44:38] :D [00:44:43] yay [00:44:54] I think I need to get some bits added to our base jessie build to make it smoother [00:44:59] * YuviPanda nods [00:45:02] shouldn't be too hard [00:46:22] jessie doesn't come out of the box with memory cgroups enabled and it takes grub changes to get it running [00:46:43] bd808: valhallasw`cloud so I suggest that we do the b/c stuff first, in the following steps: 1. complete migration to webservice-new, 2. add a k8s provider for it, 3. allow people to switch manually to the k8s backend via a commandline flag, 4. make it the default at a set date [00:46:56] bd808: oh yeah, I need to do that for k8s too [00:47:04] bd808: can you file a bug about it? [00:47:11] sure [00:47:45] webservice-new is https://github.com/wikimedia/operations-software-tools-webservice [00:48:03] which changes the current ad-hoc method of starting webservices (shell / python scripts that call each other) [00:48:06] into a real module [00:52:21] 6Labs, 10Labs-Infrastructure: Enable memory cgroups for default Jessie image - https://phabricator.wikimedia.org/T122734#1912367 (10bd808) 3NEW [01:24:39] 6Labs, 10Labs-Infrastructure: Could not remove role::phabricator::labs from rcm-3 - https://phabricator.wikimedia.org/T122733#1912392 (10scfc) What do you mean by "nothing happens"? Do you get a confirmation "Sucessfully did X", but the configuration stays the same? Do you get a blank page? Does `role::phab... [02:56:37] 6Labs, 10Labs-Infrastructure: Could not remove role::phabricator::labs from rcm-3 - https://phabricator.wikimedia.org/T122733#1912414 (10Luke081515) I took some screenshots: At first, I use configure: {F3193526} and uncheck this role: {F3193527} (This are all roles I have, and only this one is enabled) After t... [03:15:39] !log purged varnishs on deployment-cache-text04 [03:15:40] purged is not a valid project. [03:16:19] YuviPanda, bd808 ^ which should i use? [03:16:48] yurik: !log that in #wikimedia-releng [03:17:00] beta cluster !== labs [03:17:00] thx :) [03:26:07] 6Labs, 10Labs-Infrastructure: Could not remove role::phabricator::labs from rcm-3 - https://phabricator.wikimedia.org/T122733#1912425 (10scfc) I can confirm this for `toolsbeta-webproxy-01` and trying to uncheck //all// Puppet classes; unchecking individual ones work (except for the last one). Maybe this is r... [03:27:39] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912427 (10scfc) [05:35:52] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912458 (10Krenair) >>! In T122733#1912425, @scfc wrote: > `wikitech` claims that it has "successfully modified" the instance when it hasn't... [05:54:18] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912459 (10Krenair) Ah, no, actually, after playing around with the code live on silver, I think I see what's going wrong here. [06:04:05] YuviPanda: I think I'm liking this -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/NewPortals [06:04:36] the second level pages need a lot more work [06:04:36] oooo! [06:04:38] nice [06:04:40] yeah [06:04:59] maybe change ordering to 'most likely' to 'least'', which IMO is ToolLabs > Labs > Wikitech [06:05:10] true [06:06:59] bd808: what're your thoughts on the current tools 'flow', so as to say? from signup to docs to adding/removing maintainers to the home page, etc... [06:07:18] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure, 5Patch-For-Review: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912463 (10Krenair) a:3Krenair [06:07:21] or rather [06:07:28] do you have strong opinions on it already? :D [06:08:55] I think it all needs some polish. The process today looks and feels like it was built up by people with deep technical knowledge [06:09:24] The main tools page needs a big overhaul [06:10:36] There are many better "front door" pages on https://phabricator.wikimedia.org/T115650 [06:11:16] Hay's directory is the right direction to look in I think [06:11:32] * YuviPanda nods [06:12:44] And we should make it so that filling out the bits for the toolinfo.json is part of making a new tool [06:13:22] yeah [06:13:29] incl' license info [06:13:33] and offer to create a git repo [06:13:37] as part of new tool creation too [06:13:38] yes! [06:13:49] a git repo with a LICENSE and .gitignore, even [06:14:17] for some reason some part of my brain said 'you can do that with SMW' [06:14:20] and someday an optional template tool [06:14:23] that part has been summarily shot and executed now [06:14:38] bd808: yeah, +1 [06:14:44] SMW can do a lot of cool things actually [06:14:52] but ours is not in good health [06:15:27] the wiki isn't a universal solution IMO either however [06:15:33] * YuviPanda nods [06:15:40] 'tis a hammer! [06:16:01] gotta be careful with hammers, we can end up brushing teeth with it just because we technically can... [06:16:54] bd808: on the subject of template tools, https://blog.openshift.com/part-1-from-app-to-openshift-runtimes-and-templates/ [06:17:22] (along with actual template-repos too, ofc) [06:17:58] bd808: hopefully we can find some time during devsummit to actualize our learnings and inform our strategy going forward for the tools flows [06:18:35] I think we may be able to pencil in time for some outside the box thnking [06:18:54] * bd808 knows he won't sleep for 6 nights in a row [06:19:18] teehee [06:20:39] I want to think about breaking the tools help up into logical chunks so we can have a high level index like the wikitech one -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/NewPortals/Wikitech [06:20:56] * YuviPanda nods [06:21:22] maybe in terms of 'services it provides that you can use!', each of which has a troubleshooting/faq section [06:21:29] yeah [06:21:34] jobs, webservices, cron, redis, db [06:21:42] and a 'getting access' one too, possibly [06:21:56] there's a lot of irrelevant detail in there [06:22:05] I at some point in the past removed all references to toolsbeta from it [06:22:11] (I think, at least) [06:22:15] "almost all" [06:22:20] yeah [06:22:23] I'm sure I misseed a few [06:22:29] hitting edit button on that page is pretty depressing [06:24:21] There are some shell request things I want to do for wikitech soonish too. New namespaces and allowing beta features to run there [06:24:30] * YuviPanda nods [06:24:36] it's a very neglected wiki [06:25:05] yeah. It was just docs for techops and then stuff got grafted on [06:25:17] yeah [06:25:20] which is fine, that's how things grow [06:25:34] but it's time to weed the garden [06:26:25] * YuviPanda nods too [06:29:15] o/ off to stock up on sleep before I get to SF [06:30:11] I should too [06:30:35] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure, 5Patch-For-Review: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912464 (10scfc) So looking at the code you mean that the bug always existed, but noone encountered it? (Which would b... [06:33:05] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure, 5Patch-For-Review: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912466 (10yuvipanda) Yeah, and removing all puppet classes is also only very recently possible since we used to includ... [06:36:43] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure, 5Patch-For-Review: Cannot remove all Puppet classes from a Labs instance - https://phabricator.wikimedia.org/T122733#1912467 (10scfc) Ah, yes, that was in LDAP. Makes sense. [13:19:14] YuviPanda: nfs seems really slow again :/ [14:40:47] 6Labs, 10Tool-Labs: labs NFS slowness / high load - https://phabricator.wikimedia.org/T122743#1912588 (10valhallasw) 3NEW [21:33:23] (03PS1) 10Florianschmidtwelzow: Remove $wgCopyrightIcon [labs/incubator] - 10https://gerrit.wikimedia.org/r/261997 [23:06:15] Can someone review [23:06:20] https://gerrit.wikimedia.org/r/#/c/261906/? [23:06:46] I need it for my future work