[09:16:41] godog: finally tracked down a subtle gotcha that was screwing up sharing a pontoon env: https://wikitech.wikimedia.org/w/index.php?title=Help%3AStandalone_puppetmaster&type=revision&diff=1876680&oldid=1872171 [09:17:37] kormat: hah! fascinating, thanks for tracking that down [09:20:04] the only disappointment is that it turned out not to be marostegui's fault. oh well, there's always next time. [09:24:29] ye olde special-casing usernames and change behaviour accordingly [10:05:52] hello :) I could use some puppet patches to be merged for CI. Would anyone can share an hour with me this afternoon to process them please ;] [10:06:25] they should be rather straighforward (a config change for ci, some templates converted to files and an apache config change) [10:14:56] looking for volunteers to put a +1 sigil on https://gerrit.wikimedia.org/r/c/operations/puppet/+/618719 [10:40:57] godog: looking [10:41:03] I'd need npm installed on deneb to build a package (Hue) that of course pulls a lot of goodies from npm and pypi [10:41:17] is it something outrageous or not? :D [10:41:18] volans: thanks! I've merged in the meantime [10:41:34] godog: the IPv6 reverse seems not correct in DNS [10:42:07] no wait, might be pebcak [10:43:36] elukey: I usually build within a cowbuilder, to make sure build-deps are correct [10:45:29] godog: ideally I wanted just to build the upstream "build" dir separately and then copy it to a debian binary only deb, this is why I was asking [10:45:44] but I can try to see if I can make it work with a more structure build yes [10:46:22] in this case, I should add the procedure to pull all the deps into rules right? [10:47:00] FWIW it wasn't an opposition to npm on deneb, as in we've done the same before [10:47:18] but yeah build-deps in debian/control should be enough to have cowbuilder pull in the packages [10:54:27] basically what I was trying to avoid (taking a shortcut I know) was to properly package hue's deps, since the "build" dir that is needed to run it is around ~500MB [10:54:45] (so avoid pushing 500MB to gerrit basically) [10:55:46] in https://gerrit.wikimedia.org/r/c/operations/debs/hue/+/618728/ I created a bare min debian config, with the aim of building each time on deneb the horro..ehm the nice dependencies of Hue [10:56:49] (so build separately, copy the deps into the debian repository, build but not push everything to gerrit) [10:57:06] if it is not a good idea I'll drop it in favor of something better [10:59:01] yeah I think that's acceptable in this case, I guess my point is that you can still download the deps and the upstream release within cowbuilder, but either way seems fine [11:01:12] ahhh okok no idea that I could have done it, will try! [11:01:40] I thought that choosing cowbuilder would have meant having everying in the repo etc.. [11:01:43] * elukey ignorant [11:12:13] Am I okay to just run wmf-auto-reimage for a hostname where the physical machine has been replaced or are there additional steps required? The serial/tags have been updated in netbox [11:13:01] hnowlan: was the old one decommissioned with the cookbook? [11:14:35] volans: not certain - it went down with a hardware failure so I would guess not. How do I verify that? [11:15:25] if I may, why using the same hostname? [11:15:35] can't be just last NNNN+1? :D [11:15:54] and given we're at it, which hostname? [11:16:00] to see if it's still in puppetdb or long gone [11:16:28] Good point, I don't see why it can't just be another unless there's some kind of cassandra/restbase magic related to the hostname (it's long been removed from cassandra etc). It was restbase2009 [11:17:23] the hardware itself has been swapped in netbox afaics [11:17:36] if it's a replacement under warranty might make sense from dcops perspective [11:18:08] although Purchase date April 22, 2016 seems weird we use it to calculate warranty expiration (+N years) [11:18:45] It was a spare server that just got swapped in rather than a warranty replacement, old one was out of warranty [11:18:46] so I don't know the answer from te accounting PoV [11:18:49] ah [11:18:57] context https://phabricator.wikimedia.org/T256863#6320727 [11:18:57] ahhh ok [11:23:55] hnowlan: I'm checking a couple of things [11:38:07] hnowlan: I can ssh into it [11:38:07] 11:38:02 up 9 days, 21:12 [11:39:26] huh. [11:40:58] I am very confused. It's puppetised and in the cluster [11:42:04] our automation is *that* good :-P [11:42:26] * volans hides [11:43:03] :D [11:45:30] I'll leave it to you to decifer what happened :) [11:45:40] but happy to answer any reimage question you have anyway :) [11:46:30] thanks! [12:25:16] volans: for reasons that are no longer especially clear to me, i decided that implementing my own deb build script was the right thing to do. so... https://phabricator.wikimedia.org/P12189 [12:32:55] mm. it assumes it's a 'native' package (i.e. that the debian dir is in the main branch) [12:52:09] kormat: lol, fair enough :) [16:51:40] mutante: if you're around can I bother you for a couple of things? [16:53:27] volans: what's up [16:54:08] just few questions wrt dns records as I'm auditing the import into netbox [16:54:49] 1/ webperf* records seems to have 5M instead of the usual 1H TTL, although they are static addresses, the CNAMEs ofc have 5M as usual. Can those be set back to 1H? [16:55:32] hi! I am looking for input on https://phabricator.wikimedia.org/T259816. discussions, opinions, thoughts are welcomed and appreciated :) [16:56:17] sukhe: you probably want to add john and brandon (I think both in vacation right now) [16:56:42] volans: yea, they can. (should i upload a patch?) that was usually done before doing a migration to fail-over more quickly [16:56:44] volans: yeah not adding them both on purpose for the same reason :) [16:57:24] :) [16:57:33] mutante: as you want I can send one and add you [16:58:20] volans: ok, let's do that and i'll merge it [16:58:31] k thx [16:58:49] then /2 is install5001, there are dns records but I can'ts ee the VM [16:58:53] is some work in progress? [16:59:43] yes, it is. i added all the new records at the same time because i knew we will need them [17:00:02] ok [17:00:24] 3/ ganeti in eqsin, host has 2001:df2:e500:101:d294:66ff:fe81:9090/64 dns defines wmnet:ganeti5001 1H IN AAAA 2001:df2:e500:101:10:132: 0:21 [17:00:33] (sorry for the bad spacing) [17:02:37] volans: that one i don't know, i think let's ask Rob and Herron [17:02:52] (per https://phabricator.wikimedia.org/T228099#5779606 ?) [17:02:54] the former is an autoconf address (ff:fe in the middle is a good hint) [17:03:27] yep [17:03:40] so some provisioning bug [17:09:51] looks like the mapped ipv6 address aren't bound on the eqsin ganeti hosts either [17:10:40] ack, just the autoconf address on both public and private interface [17:12:09] not sure if we should reimage or manually fix the IP [17:12:31] do you think you can have a look and maybe fix it? no hurry but just to be sure will be taken care of [17:12:44] for the ganeti hosts I'd say manual, since the interface/bridge config is manually done [17:12:54] unless that's been recently automated [17:13:54] volans: sure is there a task yet? [17:14:06] no, sorry [17:15:46] as long as they survive a reboot [17:25:24] ok will have a closer look at it and follow up with you volans [17:26:00] thanks a lot herron! [17:34:13] volans: does it matter for your purposes if entries use 5M or 300 ? [17:34:39] the CNAMEs? [17:34:42] those I don't care [17:34:52] the 1H it's easier if it's 1H [17:35:09] ok [18:43:05] mutante: are there docs somewhere about how to create a new component in apt? I'm looking at the reprepro docs but they assume there only two pre-existing components; looks like now there are a lot [18:44:42] puppet question: as per https://puppet-compiler.wmflabs.org/compiler1001/24330/malmok.wikimedia.org/index.html, I switched to the pdns_recursor repository from component [18:45:10] my understanding was that this would override the existing pdns-recursor installation as well, as it would try to installation the package from there and override the existing one [18:45:14] that didn't happen [18:45:22] $ /usr/sbin/pdns_recursor --version [18:45:22] Aug 06 18:43:39 PowerDNS Recursor 4.1.11 (C) 2001-2018 PowerDNS.COM BV [18:45:37] this should have been 4.3.3, the version from component [18:45:39] how do I fix this? [18:46:03] s/pdns_recursor repository/pdns_recursor package [18:46:42] do I have to set package => absent, then run puppet, then set it to present? [18:47:07] sukhe: do you care a lot about automation or are you just doing this in one place? [18:47:29] andrewbogott: I care about automation but I also want to fix this for now :) [18:47:43] so I think I want to do both [18:48:02] for now just do apt-get update && apt-get install [18:48:26] on a newly built host you'll get the newer package. Puppet doesn't replace already-installed packages as a rule, though [18:48:53] in particular, puppet isn't really aware of the fact that a newer version of the package is available now that you have an additional repo configured [18:49:28] There is a mode where packages are always upgraded to the latest version but we pretty much never use that since it can cause too many behind-the-scenes surprises [18:49:49] interesting, thank you [18:50:18] so then in that case, theoretically, it won't be an issue for a completely new install of the system [18:50:24] correct [18:50:42] if you want to verify you could manually remove the package and dependencies and then let puppet do its thing [18:50:44] see what you get [18:51:11] sometimes order of operations between installing packages and configuring apt is tricky, so I can't make any guarantees about new hosts but it /should/ work [18:51:42] I see [18:53:33] I assumed -- naively I guess! -- that puppet would simply override the existing installation, given that it had a package to install [18:57:38] andrewbogott: the reprepro page has exactly that as a TODO. "There should be plenty of examples on how to do this in the puppet repository. " -> "TODO: perhaps add here some concrete examples for documentation purposes.". I think the example are what you get when you do: [18:57:48] ~/puppet$ git log modules/aptrepo/files/distributions-wikimedia [18:58:00] for example sukhe created one [18:58:10] andrewbogott: I recently did and I have notes, happy to share [18:58:25] mutante: you mean under 'multiple versions of the same file'? [18:58:29] yes [18:58:32] I don't necessarily want to do that but maybe that's the process? [18:58:42] I'm just trying to add a new package [18:58:46] because it says "you will need to create versioned components" [19:00:20] Is best practice to create a new component for any new package? Or should I just dump it into 'main'? [19:00:22] andrewbogott: if i just add a package i don't create a new component [19:00:31] The docs imply that it would go into 'universe' but there is no universe as best I can tell [19:00:39] universe is an Ubuntu term [19:00:52] so everything just goes in 'main'? [19:03:09] it depends if it's a new package or backporting an existing package [19:03:11] main: for Wikimedia native packages, as well as Debian/Ubuntu packages that have had source-modifications [19:29:22] there is another way to make puppet install the newer package version. the 'ensure' parameter of package{} can take version strings in addition to 'present' and 'absent'. but did not check if you can with package_from_component as well [19:30:05] mutante: thanks [22:56:38] we're planning a second DC in the EU? :o [22:59:30] yes :) [23:07:09] nice. :-) because esams is handling so much traffic? [23:15:06] yeah, esams is huge, and it's both a reliability problem and a latency problem when it's down for maintenance, and AIUI the new site will be closer to a whole bunch of submarine cables from Africa and the Middle East [23:19:33] got it - cool things planned. great! [23:23:11] (and to be clear, it will likely be many months before we are actually working on it, there's a bunch of planning and even budgetary stuff to do first) [23:23:14] but yeah :)