[09:18:08] <_joe_> feedback is welcome on https://grafana.wikimedia.org/d/RIA1lzDZk/xxx-joe-appserver [09:19:08] <_joe_> fun fact, we receive more OPTIONS requests than HEAD requests [10:22:45] * _joe_ begs for a review of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526469/ - it will allow to run the CI tests for puppet locally using the same docker container used by ci [10:23:10] <_joe_> it was written a bit in a rush, so some ways of handling docker might not be optimal [10:23:23] <_joe_> akosiaris, fsero you might have better ideas in that respect [10:23:52] <_joe_> but I think this is really useful [10:24:16] if it works... ship it! [10:24:20] I can test it if you want [10:28:08] _joe_: i have a couple of comments but testing it locally first [10:28:12] and thanks for it [10:38:55] https://www.irccloud.com/pastebin/NUA7coYo/ [10:39:00] _joe_: does it work for you locally? [10:39:25] <_joe_> fsero: it does yes [10:39:59] <_joe_> interesting [10:40:06] <_joe_> it worked yesterday, let me test now [10:45:30] <_joe_> fsero: that has nothing to do with that script btw, but I'm going to modify it quite a bit [10:51:38] _joe_: 0.5.2 works [10:51:43] so nothing to do with your script [10:51:46] added a comment [10:52:18] it also misses a log volume for getting logs [10:53:09] https://www.irccloud.com/pastebin/aPCCj9K4/ [10:57:30] <_joe_> fsero: 0.5.2 doesn't perform the local build :D [10:57:47] <_joe_> that's why you get that error [10:58:17] <_joe_> 0.5.3 doesn't work because it tries (correctly) to rebuild the gemfile [10:58:34] <_joe_> I will work it out as soon as I'm done modifying the script [11:02:37] <_joe_> fsero: looking back at your first paste [11:02:42] <_joe_> Warning: the running version of Bundler (1.13.6) is older than the version that created the lockfile (1.16.4). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [11:02:46] <_joe_> err sorry [11:03:08] <_joe_> you have defined ZUUL_URL and ZUUL_PROJECT, they are irrelevant [11:03:21] <_joe_> but I don't get the same error as you at all [11:03:33] <_joe_> what is the HEAD of your working tree right now? [11:03:47] <_joe_> is it possible you have a very outdated copy? [11:04:29] <_joe_> or, can you try again now? something tells me the problem could've been with rubygems.org [11:05:42] hi all i would like to re-attempt the change that caused the puppet errors this morning. would appreciate any rewview on the reviced plan https://gerrit.wikimedia.org/r/c/operations/puppet/+/526645 [11:07:48] <_joe_> jbond42: yeah hiera backend changes are always painful [11:08:09] jbond42: trailing spaces in commit message :-P [nitpick] /me hides [11:08:52] yes indeed in hind sight im not sure why i thought it would be hasle free :) [11:09:04] <_joe_> jbond42: IIRC, you might even need to restart apache on the puppetmasters to pick up the change, as I'm not sure the code change is picked up by the application, so only new passenger children will have the new backends [11:09:23] <_joe_> jbond42: that tricked me too many times already :P [11:10:00] _joe_: thanks i have added that to the process [12:51:33] _joe_: i did a git pull and was at latest change at the time trying again [12:53:59] <_joe_> fsero: in that case, it would be pretty strange if it would fail that way [13:12:01] _joe_: if you remove the .bundle directory from your directory where your puppet clone lives [13:12:04] you will reproduce it [13:12:16] running bundle outside container and running th container then works [13:12:24] so an issue not related to the container or your script [13:12:35] but something that should be added in the doc of the script maybe? [14:28:23] serviceops: FYI - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526685/ - probably a little bit later today, changes resolv.conf on canary api/appservers to anycast recdns (fairly well-tested and already in use for other critical clusters, but still!) [14:39:10] <_joe_> fsero: no that's actually pretty bad and it should be done in the container, I did an error there [14:39:15] <_joe_> thanks for pointing it out [15:00:35] We're going to start replacing cr1/2-codfw router's routing engines, no impact expected - T226422 [15:00:36] T226422: update RE-S-X6-64G-S in cr[12]-codfw - https://phabricator.wikimedia.org/T226422 [15:17:47] <_joe_> any ruby, or better bundler experts here? [15:18:36] <_joe_> I wanted to copy a bundle from dir A to dir B, but apparently copying gemfile, gemfile.lock and .bundle/ is not enouth [15:19:42] <_joe_> i'm now trying with bundle package / copy vendor / bundle install --local, but no luck either [15:41:38] question, is the checkbox on jenkins that says 'keep me signed in' just to feel good or what? [15:43:13] "for the next 45s" [15:43:21] XD [15:58:20] <_joe_> fsero: I found the issue you were having [15:58:24] also is very usable that if you use tabs you have to go there, select it and get back to the login button [15:58:33] <_joe_> you had a Gemfile.lock file but you removed your .bundle directory [15:58:49] Possibly [15:59:24] anyone planning a random host reimage/install in the next few hours? [15:59:26] <_joe_> anyways, if you use IMG_VERSION=0.5.4 it should work [15:59:39] <_joe_> bblack: tomorrow morning EU time is ok? [16:00:17] _joe_: if nobody's doing one Soon, I'll probably just reimage a random cache node today. I just need a sanity-check that my install_server change doesn't break installs. [16:10:03] _joe_: IMG_VERSION thing ok to merge? [16:10:14] <_joe_> bblack: yeah sorry [16:10:21] ok done [16:10:23] <_joe_> it's for a local script and I forgot :/ [16:59:50] phabricator report mails will get a new "List open tasks which have had the same assignee for four years or more" called "cookie-licked tasks" by the bug wrangler [17:00:55] nice! [17:04:04] bblack: reinstall cp1008 canary host with stretch or too early? [17:04:28] i see a comment it gets replaced soon by cp1099 [17:04:44] so maybe that is a good candidate to test the install_server [17:05:35] well..whether jessie or stretch is separate [17:13:50] oh is it still jessie? I had forgotten about it mostly heh [17:14:07] but yeah, could use it! [17:15:28] yep, its [17:15:37] jessie [18:04:53] bblack: reimage failed because the first puppet run failed for some reason. host is up though and install-console cp1008.wikimedia.org from puppetmaster works. i haven't manually started puppet