[04:57:47] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: "Package 'ethtool' has no installation candidate" - https://phabricator.wikimedia.org/T84967#935607 (10Krinkle) 3NEW [04:59:18] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: Package 'python-redis' has no installation candidate - https://phabricator.wikimedia.org/T84968#935614 (10Krinkle) 3NEW [05:02:12] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: "Package 'ethtool' has no installation candidate" - https://phabricator.wikimedia.org/T84967#935622 (10Krinkle) [05:02:13] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: Package 'python-redis' has no installation candidate - https://phabricator.wikimedia.org/T84968#935621 (10Krinkle) [05:06:44] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: Various package have no installation candidate - https://phabricator.wikimedia.org/T84967#935625 (10Krinkle) [05:09:46] 3Labs-Team, Wikimedia-Labs-Infrastructure: Puppet failure on new instance: Various package have no installation candidate - https://phabricator.wikimedia.org/T84967#935627 (10Krinkle) After logging in, however, looking at `dpkg --get-selections | grep -v deinstall` it seems these pages are installed. And he diam... [05:11:38] why does commonswiki_p not have a 'text' table? [05:12:43] we haz text? [05:16:37] abartov: comets: There is no such table in production. text is stored in external storage because it's much too large to store in a simple table and replicate. [05:17:17] abartov: comets: Text can be received from api.php prop=revisions, or from index.php?action=raw, or from xml dumps (which are stored in labs as well) [05:19:30] Krinkle: ah, I see! Thanks. So for a tool running on labs, the only way to retrieve a revision's text are API or trawling through XML dumps? [05:19:50] abartov: I believe so yes. [05:20:04] Krinkle: okay, thanks! [05:20:07] abartov: Note that the search engine is quite good these days with elasticsearch. [05:20:23] abartov: What's your use case / tool at the moment? I'm always curious to learn what people are building [05:21:07] No concreteuse case, actually. I found myself creating some Ruby classes around MW data entities, so sought to wrap text as well, for completeness. [05:21:28] Krinkle: but the actual tool I'm building doesn't need text at all. [05:21:48] Krinkle: it's an image-by-interwiki suggester for whole Commons categories. [05:22:16] abartov: I'd suggest trying to avoid working with the raw text as much possible and not encouraging it as part of a generic tool. [05:22:25] Krinkle: understood. [05:23:05] Krinkle: (probably worth mentioning somewhere on wikitech) [05:23:09] but if you or a user needs it, best to ask on labs-l or wikitech-l and see if anyone can help. There are other routes depending on what the use case is. [05:23:15] Yeah [05:23:54] I mean, maybe it's there. But some good-faith searching found no explanation. I'm sure it's well known to regular devs. This is the first time I'm trying server-side tool building rather than client-side API tools. [06:19:10] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#935670 (10Krinkle) 5Open>3declined Instance has been deleted and re-created. Can't reproduce this issue. [06:22:01] (in case this is a more appropriate channel than #wikimedia-dev ; pardon the duplication if you are in both...) [06:22:01] Who has access to the WMFlabs.org databases (specifically the one for the Wikidata-game, which I think is: 'merge_candidates' , 'p:tools-db' )? I'm trying to look into the broken stats page in the Wikidata-game, and I think it's probably a missing row in one of the tables... [07:04:48] FWIW, here's the ticket for the broken stats page in the Wikidata game: https://bitbucket.org/magnusmanske/wikidata-game/issue/168/statistics-once-again-not-displaying [07:51:12] !log wdq-mm switched wdq.wmflabs.org to project wdq-mm, load balancing between wdq-mm-01,02 instances [07:51:14] Logged the message, Master [09:16:58] 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#935862 (10yuvipanda) To recap how php things are currently run: 1. Each webservice is run as a different tool user 2. `lighttpd` + `php-cgi` is used to run things. The lighty config has somewhat saneish defa... [09:53:37] 3Tool-Labs: Create a static file server service for tools - https://phabricator.wikimedia.org/T84982#935908 (10yuvipanda) 3NEW a:3yuvipanda [09:54:44] 3Tool-Labs: Make DynamicProxy be able to proxy back to non-http protocols - https://phabricator.wikimedia.org/T84983#935916 (10yuvipanda) 3NEW a:3yuvipanda [10:05:58] 3Tool-Labs: Add --server=hhvm to webservice2 - https://phabricator.wikimedia.org/T84984#935938 (10yuvipanda) 3NEW [10:21:54] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#935970 (10hashar) 5declined>3Open >>! In T76250#935670, @Krinkle wrote: > Instance has been deleted and re-created. C... [10:22:45] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#935972 (10hashar) [10:33:10] 3Tool-Labs: Create a static file server service for tools - https://phabricator.wikimedia.org/T84982#936011 (10yuvipanda) This should probably live in tools-static.wmflabs.org ad a cookieless domain serving public_html/static by default under /toolname [11:21:17] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: Create labs project for CI disposables instances + OpenStack API credentials - https://phabricator.wikimedia.org/T84988#936061 (10hashar) [11:24:33] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: Figure out how to dedicate baremetal to a specific labs project - https://phabricator.wikimedia.org/T84989#936078 (10hashar) 3NEW [11:54:48] YuviPanda|brb: do you also do labs-morebots maintenance? :> [11:55:56] valhallasw`cloud: no :p [12:00:36] !log tools created tools-static, static http server [12:00:42] Logged the message, Master [12:03:31] meh, might need to do some pep8'ing :< [12:15:17] !log help [12:15:18] I am a logbot running on tools-exec-01. [12:15:18] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [12:15:18] To log a message, type !log . [12:19:27] Logged the message at http://wikitech.wikimedia.org/wiki/Server_Admin_Log/whatver, master [12:19:34] ok, reasonable clients ignore the , =p [12:35:04] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:45:03] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [13:09:54] Coren: andrewbogott_afk hmm, wdq-mm-02 just shut itself off into SHUTOFF state. [13:09:58] and it was on virt1011 [13:11:17] !log wdq-mm manually started wdq-mm-02 from virt1000, after it went to SHUTOFF again [13:11:20] Logged the message, Master [13:18:41] valhallasw`cloud: ping? [13:18:45] YuviPanda: sup [13:19:27] valhallasw`cloud: so, I’m setting up tools-static atm. static http file server. [13:19:32] I see [13:19:41] valhallasw`cloud: thinking of poking around on gerrit-patch-uploader to see if I can switch that over. [13:19:43] can I? [13:19:52] it does have ssl, etc [13:21:00] valhallasw`cloud: heh, I see most of your stuff is in the ‘static’ tool :) [13:22:47] YuviPanda: yeah, sure [13:23:04] YuviPanda: jquery etc is in static [13:23:34] valhallasw`cloud: yeah, and static is now available in http://tools-static.wmflabs.org/static/ [13:23:56] YuviPanda: you're now in gerrit-patch-uploader [13:24:06] valhallasw`cloud: \o/ cool. in github too? :) [13:24:32] github should have a function to add people as default co-owners :+ [13:24:37] awww [13:24:53] yeah, you're in [13:25:39] cool [13:55:36] valhallasw`cloud: I’m going to deploy https://github.com/valhallasw/gerrit-patch-uploader/commit/9c7c528c5b0075ba09f3028bc3c7c65675e8a917 now [13:55:45] YuviPanda: fab it! [13:55:50] oh [13:55:52] there’s fab? [13:56:04] no just a crappy ./deploy.sh :-p [13:56:12] I mean domerge [13:56:18] which is not in git for some reason :| [13:56:26] heh [13:56:39] I guess fab will ssh in, do a git pull, and then restart webservice [13:57:05] https://www.irccloud.com/pastebin/ow0ePuFH [13:57:06] ^ [13:57:14] it also pip installs requirements [13:58:09] YuviPanda: and killing fcgis might be better than restarting webservice, but whatevs [13:58:46] valhallasw`cloud: hmm, I did it but the html doesn’t seem to have changed much... [13:58:47] https://tools.wmflabs.org/gerrit-patch-uploader/ [13:59:21] YuviPanda: I see tools-static? [13:59:25] [13:59:49] valhallasw`cloud: oh, right. I was looking in wrong column [13:59:53] valhallasw`cloud: and yay :D [13:59:57] valhallasw`cloud: it should be *slightly* faster [14:00:09] I need to set up nginx-slowfs at some point to make it much faster [14:00:21] YuviPanda: so what does tools-static serve and what not? [14:00:37] e.g. https://tools-static.wmflabs.org/gerrit-patch-uploader/static/chosen.css doesn't seem to work [14:00:52] valhallasw`cloud: it serves from $toolhome/public_html/static [14:01:01] valhallasw`cloud: and if your $toolhome is +x [14:01:16] hmkay [14:01:17] valhallasw`cloud: so if you put your static files in there, it should work [14:01:39] valhallasw`cloud: so it basically serves public_html/static as tools-static.wmflabs.org/$toolname [14:01:42] YuviPanda: doesn't follow symliks? [14:01:45] oh, [14:01:51] yeah, works [14:01:52] https://tools-static.wmflabs.org/gerrit-patch-uploader/chosen.css [14:01:52] valhallasw`cloud: https://tools-static.wmflabs.org/gerrit-patch-uploader/chosen.css [14:01:54] yeah [14:01:54] <3 [14:01:57] :D [14:02:34] valhallasw`cloud: this will also simplify things for uwsgi / hhvm [14:04:15] YuviPanda: I've also been thinking of how we could use puppet on a tool level [14:04:33] valhallasw`cloud: go on... [14:04:59] which would just declare 'git checkout here, .lighttpd.conf with x y z (might be a puppet class for this), symlink for static, index.html for if the webservice crashes' [14:06:00] (I was really thinking of just building that as 'instant flask app, just add water' kind of thing, but I think it should be more generally applicable) [14:06:02] also virtualenv [14:07:18] might even set up gunicorn! [/evil] [14:07:18] valhallasw`cloud: yeah, I’ve been thinking of similar things too [14:07:22] valhallasw`cloud: puppet is the wrong thing for this [14:07:24] valhallasw`cloud: uwsgi! [14:07:27] ;-) [14:07:32] why is puppet the wrong thing? [14:07:38] valhallasw`cloud: you need https://devcenter.heroku.com/articles/buildpacks [14:08:12] hm. [14:08:38] valhallasw`cloud: https://github.com/heroku/heroku-buildpack-python [14:08:39] for example [14:08:51] valhallasw`cloud: sets up virtualenv, installs things, sets up webservice, static links [14:08:59] valhallasw`cloud: and in our case, ‘bigbrother' [14:09:05] (but I really want to replace that with monit) [14:11:11] YuviPanda: I don't see how that's better than puppet [14:11:17] that's just Yet Another Way(TM) of doing things [14:11:59] valhallasw`cloud: no, puppet is a ‘orchestration engine' [14:12:03] valhallasw`cloud: what we want is a painless deployer [14:13:19] YuviPanda: still not sure how puppet does not fit that bill. It makes sure the environment is in some well-defined state, and makes sure services (specifically, the webservice) are in a well-defined state. [14:14:14] https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/pip-install [14:14:32] that looks horrible :-p [14:14:42] well, yeah. but the concept is sound :) [14:16:31] valhallasw`cloud: also, puppet… you need merge rights in operations/puppet :) also, it’s super heavy weight, so don’t want to have one repo per tool. *also*, it doesn’t know jackshit about GridEngine, so we’ll have to write that anyway [14:17:12] YuviPanda: not puppet-as-a-tool-which-runs-ever-minute, but puppet-as-in-what-you-run-when-you-deploy [14:17:29] the tool would just have a tool.pp manifest [14:17:52] i.e. the tool author would run puppet apply [14:18:08] valhallasw`cloud: it’s a terrible, terrible deployment tool [14:18:10] believe me, I’ve tried :) [14:18:14] ok [14:18:27] valhallasw`cloud: I spend a few days trying to get quarry deployed with puppet [14:18:30] it has worked reasonably well for vagrant for me :-p [14:18:36] valhallasw`cloud: and then gave up and wrote the fabscript in an hour or something [14:18:40] mkay [14:18:51] valhallasw`cloud: vagrant puppet isn’t a deploy thing :) [14:19:00] of course it is. [14:19:13] it deploys stuff on a bare vm [14:20:45] valhallasw`cloud: it provisions a VM to a particular state and keeps it there. [14:22:00] YuviPanda: hmmm, true. [14:22:05] YuviPanda: it doesn't really do updates much [14:22:18] yup [14:22:23] that’s also why puppet generally runs in a cron [14:33:41] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<33.33%) [14:46:36] 3Tool-Labs: Create a static file server service for tools - https://phabricator.wikimedia.org/T84982#936339 (10yuvipanda) Ok, tools-static.wmflabs.org exists now. Serves things under /$toolname anything from `/data/project/$toolname/public_html/static`, assuming that path is accessible to www-data user. Still t... [14:48:41] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [14:48:48] valhallasw`cloud: https://tools.wmflabs.org/gerrit-patch-uploader/static/chosen.css vs https://tools-static.wmflabs.org/gerrit-patch-uploader/chosen.css - tools-static seems to be almost 4x faster?! [14:48:55] is that just my local connection acting up? [14:49:55] There'd be one less leven of indirection and latency, and a simpler mechanism. Faster is not surprising - all the more so for small files where setup and latency dominate. [14:50:53] Coren: yeah, but 4x? [14:51:35] YuviPanda: it's also not clear whether that '4x' number actually means anything =p [14:51:42] Relative means nothing on short intervals. [14:51:44] Coren: anyway, this is in preparation for https://phabricator.wikimedia.org/T78783, which I guess we’ll just do fastcgi directly [14:51:46] indeed [14:51:51] valhallasw`cloud: I just did ‘time’ and a curl :P [14:51:53] not the most accurate [14:52:14] Coren: hmm, do we have a way to restart proxylistener at all? [14:52:25] we don’t, do we? unless we also are ok with restarting all webservices as well [14:53:22] I don't think we do, but the side effects from doing so are not entirely catastrophic so it can be done if really necessary (but it's undesirable) [14:53:38] Coren: right, needed for https://phabricator.wikimedia.org/T84983 [14:54:49] I still think you're doing that HHVM thing in an overly complicated way - you could simply have configured HHVM as a lighttpd fcgi backend with half a dozen lines in the config. :-) [14:55:26] Coren: am just trying to remove a layer :) plus, this is needed for uwsgi anyway. [14:55:43] lighttpd -> uwsgi just is… wrong [14:57:26] Yeah, but you're doing this at the cost of greaty increasing the complexity and "smartness" required of what was intended to be a simple, blind proxy. The more logic you pull into the proxy the more brittle every single webservice becomes. [14:57:57] Coren: the static server isn’t in the proxy [14:58:08] it’s a different domain that just serves static files [14:58:27] I know. I didn't complain about *that* either. [14:58:39] Coren: oh, it’ll be *less* smart with the proxylistener change [14:58:51] Coren: right now it prepends http:// to the backing URL. I just want it to stop doing that, that’s all. [14:58:53] (In fact, if you look at long-gone email thread you'll see that was on the todo list) [14:58:55] no other smartness required. [14:59:04] oh, which one? [14:59:28] A static web server for things like css and js [14:59:48] ah, right [15:00:06] I’m going to put a simple landing page, write up a wikitech doc and then call it done [15:01:52] But back to the previous idea: restarting proxylistener has perverse effects and needs to be schedule and announced as an outage. [15:02:06] Coren: I’m also wondering if I should build a custom nginx package, with slowfs, which would make the NFS access for the static files much faster. at some point in the far future, maybe. [15:02:28] Coren: hmm, right. I’m looking to see if there’s a way I can do this *without* restarting proxylistener, only restarting the nginx [15:02:38] well, reloading nginx config, which is much faster [15:05:05] Coren: hmm, what writes to proxylistener again? [15:05:20] Coren: oh, portgrabber? [15:05:45] yeah, portgrabber [15:52:29] Coren: I'm just checking in before I go to sleep… anything I can do to further the jessie/lvm cause? [15:53:29] andrewbogott: btw, a VM on virt1011 also went to ‘SHUTOFF’ today and had to be ressurected. Nothing urgent, will file a phab task [15:54:27] YuviPanda: when did it happen? I have 'compute' disabled on 1011 right now but instances that are already running there should be ok... [15:54:29] andrewbogott: No, but I've got a few happy fun test to run today that may well be a fix. I've got a good grasp of where to hook myself in systemd now. [15:54:41] andrewbogott: oh, about 2h ago? [15:54:48] YuviPanda: hm, ok, that wasn't me [15:54:50] weird. [15:55:15] Coren: a fix that you can contribute to upstream? Or do you no longer thing the issue is a broken lvm? [15:57:52] andrewbogott: I haven't ruled it out yet, but my current tests locally point to poor systemd configuration instead. [15:58:46] I expect to be able to tell for sure this PM. [15:59:14] YuviPanda: 1011 and 1010 are on probation again because some component there has decided to use libvirt/spice even though spice isn't installed. [15:59:35] So occasionally it's all "OMG I can't find the spice binary" and random things fail [15:59:46] andrewbogott: oh, so new instances on hold again? [16:00:01] I just did a big diff of the package versions on the two systems and… no differences that I can see. So I'm baffled. [16:00:08] YuviPanda: yeah, for the time being. [16:08:38] YuviPanda: The proxy 500s. [16:09:05] yup, fixing... [16:10:18] Hi people [16:10:32] https://tools.wmflabs.org/xtools/pages/index.php?user=Mdann52&lang=en&wiki=wikipedia&namespace=0&redirects=noredirects [16:10:34] Tools down? [16:10:51] Proxy having issues, YuviPanda is on it. [16:11:06] Qcoder00: it’s back up [16:12:14] YuviPanda: But everything from /admin/ appears effed up; incluing the errordocuments. [16:12:23] https://tools.wmflabs.org/ [16:12:36] yup, admin needs a restart. restarting now [16:12:45] apparently, modifying lua files takes effect *immediately* [16:12:49] and not on reload of nginx config [16:22:00] YuviPanda, quick labs question. There's no way to quickly spin up a single instance without doing that in a project, right? I'm trying to figure out if the option 'spin up a phab development instance' is a reasonable option for anyone who's not basically WMF [16:22:25] valhallasw`cloud: not really, no. but you can spin that up in the phab project easily [16:22:46] YuviPanda: except that means people have to request access to the phab project [16:22:54] valhallasw`cloud: yup, that. [16:22:55] YuviPanda: ok, I think 1010 and 1011 are working OK now… I'll leave them on overnight, and will review again tomorrow. [16:23:07] andrewbogott: you have them enabled for new instances? [16:23:11] yeah [16:23:30] YuviPanda: hm, ok. then the vagrant option is still more reasonable, I guess. [16:23:36] oh definitely [16:23:38] andrewbogott: yay, thanks [16:24:03] YuviPanda: this last issue really surprised me, hope there aren't others. [16:24:07] But, for now: sleep! [16:37:16] Coren: ok, so tools is in a stable state (outage lasted 2 minutes total), I’m going to get some food. puppet disabled on tools-webproxy for now [16:38:42] !log tools puppet disabled on tools-webproxy because urlproxy.lua is handhacked to remove stupid syntax errors that got merged. [16:38:48] Logged the message, Master [16:39:27] 3Tool-Labs: Create a static file server service for tools - https://phabricator.wikimedia.org/T84982#936463 (10Krinkle) https://github.com/tool-labs/static-browser/pull/4/files [17:20:36] Coren: ok, all good and puppetized now, no more live hacks :) [17:21:07] 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#936520 (10yuvipanda) [17:21:08] 3Tool-Labs: Make DynamicProxy be able to proxy back to non-http protocols - https://phabricator.wikimedia.org/T84983#936518 (10yuvipanda) 5Open>3Resolved Yay! :) [17:21:32] Joy! [17:44:34] 3Tool-Labs: Webservice restarts should be faster - https://phabricator.wikimedia.org/T85010#936563 (10yuvipanda) 3NEW [17:45:32] yay [17:47:00] annika_: ? [17:47:15] i want faster webservice restarts ;) [17:47:42] annika_: :) [17:47:56] annika_: I need to profile to what’s taking longer - GridEngine or lighttpd [17:48:03] annika_: do tell me other things you find painful about tools [17:48:21] i think it's the grid [17:48:40] tickets aren't worked on :( [17:48:41] explain? [17:48:45] annika_: which one? [17:48:49] outside of the tcl ones :) [17:48:53] :p [17:50:27] It's probably the grid itself; scheduling is done at (iirc) 5s interval. [17:50:33] how about https://phabricator.wikimedia.org/T64156 [17:50:43] i tend to forget them [17:51:12] Coren: oh, can we reduce that? [17:51:24] annika_: Small feature requests tend to be merged in much faster when they are accompanied by a patch. :-) [17:51:45] i don't speak perl, sorry [17:52:19] YuviPanda: Not usefully; it's an async process in the first place and reducing the interval will just increase the load without significant gain. [17:52:37] YuviPanda: It'd be much more useful to have a mechanism to sighup a job instead. [17:52:44] Coren: oooh, hmm, yeah. [17:53:09] Coren: so if we set up uwsgi for example, easy enough. we could even set it up to ‘autoupdate' [17:53:47] other than that it's only two tcl bugs [17:54:19] annika_: The problem is that we've got nobody with any tcl expertise in ops. Hell, I don't think we have anyone with tcl /experience/ at all. [17:55:22] what has that to do with compiling and installing software? [17:55:52] annika_: Doing a proper package requires understanding properly how the software works, where it expect stuff, etc. [17:57:17] annika_: This is why I have no problem making a package out of a perl or python module for instance - I know how to integrate those properly. [17:57:42] i think that is too much perfection [17:57:58] annika_: It's not about perfection, it's about maintainability. [17:58:12] sigh [17:58:18] annika_: But what prevents you from installing that software locally to your tool? [17:58:25] That has no packaging requirements. [17:58:41] annika_: yeah, you can compile it yourself... [17:58:45] without needing to package it [17:58:46] Most people who have one-off requirements for unusal python modules do so for instance. [17:58:47] i did [17:59:12] it's just weird not to use standard paths [17:59:31] and there is another user that uses the same packages as i do [17:59:44] they can copy it, but still … [18:00:04] annika_: Easier still would be to use your copy, provided you're careful about updates. [18:00:27] annika_: You can also put it in /shared like the pywikibot users do. [18:00:33] nice [18:01:30] But the basic rule is: if it's to be installed as part of the system (and it isn't a simple, puppetizable script) then it must be a proper package. [18:02:19] Otherwise we're entering mainetance hell, dependency woes, and stability despair. [18:02:39] hmmmmm [18:03:03] That said, if you know of a maintainer that /does/ keep a package up to date, we can import that. [18:03:19] But it has to be a proper deb [18:03:49] i'll try to maintain that myself (without packaging) [18:05:07] Feel free to create a directory in /shared/ for locally brewed tcl things you think may be of general use. [18:06:15] /shared/ is accessible to all nodes, so is freely usable for things like this. [18:06:17] http://www.bbc.co.uk/news/world-us-canada-30555997 [18:06:28] Time to watch relevant articles? [18:07:32] i should close both tickets [18:09:22] Coren: me and valhallasw`cloud were talking about monit elsewhere [18:09:25] if not for bigbrother [18:09:29] at least for arbitrary monitoring [18:09:53] like, ‘if gerrit-patch-uploader has not written to log file in last 12h, email me' [18:11:11] Coren: didn’t https://phabricator.wikimedia.org/T62864 get resolved? [18:11:51] YuviPanda: The ways of Legal are Dark and impenetrable. I dunno where they're at atm. [18:11:58] aaaaaah [18:11:59] right [18:13:54] 3Tool-Labs: Install rake for Tools Labs - https://phabricator.wikimedia.org/T70208#936604 (10yuvipanda) Is this still needed? [18:16:19] 3Tool-Labs: Upgrade awk to 4.1.1 - https://phabricator.wikimedia.org/T73273#936606 (10yuvipanda) 5Open>3declined Trusty also has only 4.0.1, and I don't think we want to maintain such a package unless there's more requests. The reporter seems to have fixed it for themselves for now, so resolving as Declined... [18:17:13] 3Tool-Labs: Web services continually restarting - https://phabricator.wikimedia.org/T71934#936609 (10yuvipanda) 5Open>3Invalid No update in about 4 months, so closing as invalid for now? [18:17:32] 3Tool-Labs: install tdbc and tdbc::mysql - https://phabricator.wikimedia.org/T53129#936612 (10Giftpflanze) 5Open>3Resolved Because it is too complicated for the Tool Labs admins to package this, I will compile it and store it in /shared. [18:17:33] 3Tool-Labs: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#936614 (10Giftpflanze) [18:17:57] 3Tool-Labs: Update Tcl 8.6 - https://phabricator.wikimedia.org/T78397#936618 (10Giftpflanze) 5Open>3Resolved Because it is too complicated for the Tool Labs admins to package this, I will compile it and store it in /shared. [18:17:58] 3Tool-Labs: Multiple webservices running for one tool - https://phabricator.wikimedia.org/T76578#936616 (10yuvipanda) Did you use webservice2 the first time or did you just use normal webservice? [18:19:03] huh, ticket dependencies [18:19:57] 3Tool-Labs: Support SPDY/3 on the proxy - https://phabricator.wikimedia.org/T67134#936620 (10yuvipanda) p:5Normal>3Low Will probably happen when we've new nginx packages available, probably from upgrade to *next* LTS, which isn't due for a while... [18:20:20] 3Tool-Labs: Support SPDY/3 on the proxy - https://phabricator.wikimedia.org/T67134#936623 (10yuvipanda) a:5yuvipanda>3None [18:20:45] Coren: shouldn't T64156 be trivial? [18:21:41] annika_: should be, but I don’t think ENV variables are the right thing to do there [18:21:49] annika_: if at all, perhaps a ~/.jsub that has default ops [18:21:51] *opts [18:22:06] really? [18:22:16] i would want them in my crontab, but not otherwise [18:22:36] you can create a simple wrapper script :) [18:22:42] nooooo [18:22:54] i can also write the options again and again [18:23:01] also, cron? I don’t think env variables would be any better than just options [18:23:03] or maybe a cron variable [18:23:42] i just think an env var is more elegant [18:23:46] but well [18:36:25] Coren: do you agree that this is a bad idea? [18:51:00] Environment variables tend to be a source of great subtle issues when they affect the behaviour of programs; I've nothing against it in principle but I don't see it as a big win. An rc file I can believe in. [19:07:16] annika_: what Coren said :) [19:11:20] 3Wikimedia-Labs-Infrastructure, Labs-Team: Puppet failure on new instance: Various package have no installation candidate - https://phabricator.wikimedia.org/T84967#936726 (10Krinkle) [19:11:36] 3Wikimedia-Labs-Infrastructure, Labs-Team: Puppet failure on new instance: Various package have no installation candidate - https://phabricator.wikimedia.org/T84967#935607 (10Krinkle) [19:25:55] 3Wikimedia-Labs-Infrastructure, Labs-Team: Puppet failure on new instance: Various package have no installation candidate - https://phabricator.wikimedia.org/T84967#936758 (10coren) Part of it is likely to be that puppet installs apt repos from which some of those packages or their dependencies are installed; th... [20:38:50] anyone used python virtualenvs in cron on labs? [20:39:06] fhocutt: yep [20:39:13] fhocutt: how's life? :-) [20:39:18] pretty good! [20:39:31] fhocutt: the easiest way is to run code using /path/to/venv/bin/python /path/to/script.py [20:39:50] cool, thanks! I will give that a try. [21:03:21] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#937056 (10Krinkle) >>! In T76250#935970, @hashar wrote: >>>! In T76250#935670, @Krinkle wrote: >> Instance has been... [21:26:03] Hi! Does anyone know what's up with glam tools? specifically glamorous? the site has been down for a few days [21:32:23] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#937096 (10hashar) >>! In T76250#937056, @Krinkle wrote: > I didn't rebooted it. But it worked fine when provisioning... [21:35:48] 3Labs-Team, operations: labs VMs install with outdated packages - https://phabricator.wikimedia.org/T85024#937097 (10Gage) 3NEW [21:36:02] 3Labs-Team, operations: labs VMs install with outdated packages - https://phabricator.wikimedia.org/T85024#937097 (10Gage) In my local setup I accomplish this by using a "mini ISO" which does not contain many packages, forcing the installer to download and use the latest versions instead of stale ones: https:/... [21:51:26] compiling is pain … [21:55:55] !fingerprints [21:55:55] ssh keys for bastion hosts: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [21:56:19] Coren: poke about the fingerprint for trusty.tools? ^ [22:23:33] OMG I am a fucking idiot. [22:23:51] have you rebooted something? [22:24:19] Yes, yes I have. I meant to issue that command on the test jessie instance I was working on - not on tools-login. [22:26:48] Thankfully the entire reboot cycle is <30s but it still sucks for people I've needlessly interrupted. Sorry. [22:27:01] Coren: to paraphrase the one about "rm -rf on the wrong directory", there are two kinds of people: those who have done "shutdown -r now" on the wrong machine; and those who have yet to. [22:27:36] Oh, indeed, I've done worse before also. It's still pretty annoying. [22:27:57] (Issuing a poweroff to a box that didn't have an ILO in a remote DC is one of my worse) [22:28:20] You can't have been a sysadmin for some 25 years without having broken a box or two on the way. :-)