[00:06:16] Coren: Hi. Any news about the work you planned yesterday about preparing a trusty machine for fcgi? Just wondering if I can use that now, [00:27:24] Coren: you around? I want to bounce an idea off you [00:37:20] nvm everybody because I solved my issue with actual "nvm" [00:38:43] https://github.com/creationix/nvm [01:46:03] Betacommand: Semi-sorta. What's up? [01:46:28] see https://bugzilla.wikimedia.org/show_bug.cgi?id=72931 [01:49:35] Sounds sane, but I'm hardly the reference in re the core; I have all of one bugfix to my name there. :-) [01:50:34] Coren: just wanted to bounce the idea off you and get some feed back before filing the bug [01:50:43] but legoktm did that [01:56:19] At any rate, sounds sane to me and I can think of a couple useful cases for spam hunting too. [01:57:32] Coren: care to drop your use cases on the bug? [07:03:05] Hoi ... the webservice of Magnus for Autolist is flaky ... http://tools.wmflabs.org/autolist/index.php [07:03:11] can someone restart it [07:03:26] I have a demo in thirty minutes and I need it to work [07:25:57] GerardM-: restarted [10:59:03] could someone please start magnustools web service ?? [11:02:20] GerardM-: done [11:05:37] thanks YuviPand [14:22:56] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c12 (10Antoine "hashar" Musso (WMF)) Created attachment 17012 --> https://bugzilla.wikimedia.org/attachment.cgi?id=17012&action=edit Raw thread... [14:24:41] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c13 (10Antoine "hashar" Musso (WMF)) From Jenkins log https://integration.wikimedia.org/ci/log/Plugins%20-%20Gearman/ ( hudson.plugins.gearman.lo... [14:31:12] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c14 (10Antoine "hashar" Musso (WMF)) The rest of the executors are hold in lock with: "Gearman worker deployment-bastion.eqiad_exec-2" daemon... [16:25:17] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c15 (10Antoine "hashar" Musso (WMF)) I disconnected the Jenkins Gearman client which is holding the lock. That unleashed the jobs that are meant... [19:36:47] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-login.diskspace._var.byte_avail.value (11.11%) [19:38:21] so meta when the monitoring server is out of disk [19:40:28] ignore that, tools-login of course [19:40:40] looking there [19:50:44] xtools down again?.. [19:50:46] !log tools - apt-get clean on tools-login, and gzipped some logs [19:50:50] Logged the message, Master [19:52:10] comets: yea, i think so, but unrelated from toollabs itself [19:52:13] "The URI you have requested, /xtools/, is not currently serviced." [19:52:36] yea what does that mean? [19:52:56] * comets takes it to a service station :P [19:53:20] i think it means we gotta contact the maintainers [19:53:39] This URI is part of the xtools tool, maintained by Tparis, Cyberpower678, and Hedonil. [19:57:02] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [19:58:53] icinga-wm: yea, nice, still confusing though that it's not actually on labmon1001 but we just put the monitor service there [19:58:59] anyways, works :) [20:52:26] hmm, backend is overloaded; [21:29:44] 3Wikimedia Labs / 3deployment-prep (beta): API error: unable to connect to redis - 10https://bugzilla.wikimedia.org/72982 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None Browser tests on beta labs have been getting a fair number of these in recent times: Exception Caught: Unable to connect to re... [22:26:44] YuviPanda, bd808 : whoever implemented `labs-vagrant git-update` o/ <3 [22:26:54] spagewmf: that's yurikR1 [22:26:57] I think [22:27:02] I did it [22:27:05] :p [22:27:07] Where did the R come from [22:27:19] bd808: oh? [22:27:22] but yurikR1 has been making some adjustments to tha backing script [22:27:33] damn, I've been crediting the wrong person all the time [22:27:40] * YuviPanda buys bd808 a whiskey instead [22:27:40] I hooked it up to labs-vagrant [22:27:51] ah, right [22:27:55] * YuviPanda hasn't touched that in a while [22:28:01] * bd808 knows [22:28:09] too many toys [22:28:11] I should start getting more organized about opsy work now I'm actually in ops... [22:28:19] bd808: yeah, totally :| [22:28:21] well, thanks. Makes labs-vagrant more pleasant [22:28:36] spagewmf: yw. I did it because you asked [22:28:53] bd808 make me a sandwich [22:29:04] hmm, now that we have hiera in wikitech, I wonder if labs-vagrant should use it in some form [22:29:16] spagewmf: you forgot the magic word ;) [22:29:24] YuviPanda: yes it should [22:29:33] I'm not fully sure in what form... [22:29:45] perhaps a JS gadget or something... [22:29:49] ok that sounds terrible [22:30:02] YuviPanda: to allow the per-host customizations to come from the yaml backend [22:30:11] I thought "hiera" was https://www.mediawiki.org/wiki/Extension:Hiero , but probably not [22:30:21] bd808: role list, etc? [22:30:59] is WMF going to hiera Yuvi's brother? :) [22:31:00] YuviPanda: The bits that can be changes that don't have a UI in the labs-vagrant command. Like hostname mappings and other tweaks [22:31:10] aaaah, right [22:31:36] But I'm still trying to figure out how to make them discoverable [22:31:48] without resorting to "read the puppet code" [22:32:11] yeah [22:32:20] I started working on a bit for the Vagrant plugin to "get help" on roles [22:32:44] but it didn't go very far yet. It does pretty print the comment headers via rdoc parsing [22:33:22] `vagrant roles info centralauth` is the prettiest [22:34:14] I'd like to find a way to use the actual ruby code from the plugin in labs-vagrant too. [22:34:44] Maybe I need to take the plunge and write a plugin to have Vagrant manage the local host directly [22:35:00] It might not actually be that tricky [22:35:15] do it! :D [22:35:26] but along with the 40 other things you are doing, I guess... [22:35:32] yeah :) [22:35:37] too many toys [22:36:01] I have books to read, lego sets to assemble, code to write, dogs to play with .... oh yeah and a job [22:36:21] :D [22:36:25] :) [22:36:30] bd808: just don't implement `vagrant -f destroy` :) [22:36:34] yeah, and being by pretty mountains is also destracting [22:36:36] *distracting [22:36:57] marxarelli: That's the easiest though -- sudo rm -rf / [22:37:01] on the plus side, merged a number of puppet patches! [22:37:06] * YuviPanda goes to sleep on the afterglow [22:38:01] spagewmf: I found the commit where I credited you -- https://gerrit.wikimedia.org/r/#/c/161360/ [22:50:57] 3Wikimedia Labs / 3deployment-prep (beta): API error: unable to connect to redis - 10https://bugzilla.wikimedia.org/72982#c1 (10Greg Grossmeier) Chris: can you see if this is related (or not) to runs of scap on Beta Cluster a la bug 72366? 503s are now suspicious to me :) [22:51:11] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366 (10Greg Grossmeier)