[09:53:04] is it possible to download files from the net inside pbuilder on deneb? [09:56:04] it is yes [09:56:16] I had to do it for some packages [09:56:26] elukey: ah. do you have an example i could look at? [09:56:51] I added some info to https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/DebianPackaging [09:56:56] :) [09:57:15] ah hah! [09:57:31] elukey: ignore what they say about you, you're actually useful! 💜 [09:57:37] ahhahahaah [09:57:54] kormat: although very likely you're doing something that will make moritzm very angry [09:58:18] ema: this is a fact of life [09:58:29] the context is https://phabricator.wikimedia.org/T266763 [09:58:49] I know that the positive thoughts from kormat will last only few hours [09:59:03] elukey: yeah, enjoy it while it lasts! :) [09:59:11] :D [09:59:26] the only way around it i can think of is to install golang-go from backports, and download all the go dependencies 'manually' inside pbuilder [10:01:25] but it is really tedious [10:06:03] kormat: are you sure they actually use go 1.14+ features? [10:06:30] jayme: it _builds_ with go 1.11 and go1.13, at least. but i don't really want to run it with a version older than they "officially" support [10:06:51] if it builds, you're fine :D [10:08:14] besides, if they use a go1.14 feature in the next release, i'd have to redo a whole bunch of stuff then anyway [10:08:59] kormat: for go dependencies what we did a lot is patch the upstream tarball with a up-to-date vendor dir before importing with gbp (see: https://wikitech.wikimedia.org/wiki/Helm) [10:10:38] Sure, maybe you need to upgrade/build/backport golang 1.14+ at some point (if they use a feature). But you could decide to worry about that when the time comes (and if) [10:10:52] jayme: unfortunately orchestrator doesn't use go modules yet [10:12:05] ouch...what do they use then? [10:12:07] it's old-school, with _some_ dependencies vendored, and some not. [10:12:14] jayme: custom build scripts \o/ [10:13:04] oh, wow...thats...unpleasant [10:13:29] but still...you can probably vendor manually somehow [10:13:47] ah. and i just noticed that dh-make-golang strips `vendor/` when importing the tarball. lovely [10:16:47] jayme: i'll play around with it and see if i can figure something out. thanks for the pointer :) [10:17:20] yw, have fun! :) [10:17:44] * jayme continues with his own go build mess [10:18:18] I am still expecting someone to somehow shoehorn autoconf/automake into a pure Go project [10:26:39] klausman: ಠ_ಠ [11:00:28] <_joe_> klausman: don't give chris ideas please [11:01:18] <_joe_> kormat: my general advice is to ignore whatever dh-make-golang does with dependencies and vendoring [11:01:33] _joe_: ack. that's the conclusion i came to [11:01:35] <_joe_> I usually just copy their debian dir into a git tree with all the dependencies vendored [11:01:52] <_joe_> that's exponentially better than fetching them at build time from the internet [11:01:57] <_joe_> at least they're auditable [11:03:53] possibly even reproducible! [12:05:48] <_joe_> klausman: well let's not get ahead of ourselves here [12:24:46] A man can dream [12:30:16] ha, seems like a lot of us are having the same dream [13:59:16] how do i get pbuilder to install a package from backports? [13:59:33] BACKPORTS=yes [14:00:30] README.md in modules/package_builder also has a list of other available hooks [14:01:19] cdanis: i've tried putting that in .pbuilderrc, and in the gbp buildpackage env, but neither seems to work [14:04:33] ah, correction. `BACKPORTS=yes gbp buildpackage` definitely does include backports, but apt still doesn't install golang-go from it, even though i've requested `golang-go (>=2.14)` [14:05:49] https://phabricator.wikimedia.org/P13104 [14:09:42] kormat: you're requesting a future version [14:09:56] '2:1.14~1~bpo10+1' is < than '2:1.14' in debian terms [14:10:12] oh ffs [14:10:22] you can use apt_pkg.version_compare [14:10:24] in python [14:10:36] I'll send you the few lines in query [14:10:45] or dpkg --compare-versions $A gt $B in the shell [14:11:05] ok. so.. `(>2.13)` should work? [14:11:14] no, that's not right either [14:11:31] http://lira.no-ip.org:8080/doc/python-apt-doc/html/library/apt_pkg.html#apt_pkg.version_compare [14:14:13] i've changed to `2:1.14~0`, which is apparently less than the available version [14:14:21] but it doesn't fix the issue [14:14:28] ` pbuilder-satisfydepends-dummy : Depends: golang-go (>= 2:1.14~0) but it is not going to be installed` [14:15:30] normally you'd have to do `apt-get -t buster-backports install golang-go` to do this manually [14:15:35] but i've nfc how to get pbuilder to do that [14:17:48] when the bulds fails on deneb, it spawns a shall via a pbuilder hook, does dpkg -l show the 1.14 version being installed in the build environment? [14:18:21] it's not giving me a shell, it's exiting clean out [14:19:26] tryring with SHELL_ON_FAILURE=yes just in case [14:20:05] ahh [14:20:10] i think it's actually working now [14:20:32] looks like i dropped BACKPORTS=yes when i re-ran it after changing the version constraint [14:21:30] `Setting up golang-go (2:1.14~1~bpo10+1) ...` [14:21:48] can BACKPORTS=yes be specified in some config file? [14:23:27] not in the current setup without adding a layer around it, it get passed to the pbuilder build only via the exported env var [14:23:42] uff. ok, thanks [14:24:25] kormat: I have a small bash script for various packages I rebuild often fwiw [14:24:30] you can poke in my home on deneb [14:24:34] *-release [14:24:42] i've seen your 'small' scripts before [14:24:58] lol, like which one? :D [14:25:12] ~volans/spicerack-release in fact [14:25:17] we could just add a small wrapper like "wmf-trigger-build" which reads a .settings file or so and then injects the necessary hooks [14:25:48] I agree some conff file would be nice to not be required to know which settings a build needs [14:25:52] <_joe_> kormat: you could add a BUILDING.md file :) [14:25:59] <_joe_> moritzm: you mean... CI? [14:26:05] <_joe_> 😱 [14:26:15] <_joe_> we don't want modernity here! [14:26:27] <_joe_> we craft our debian package by hand [14:27:17] our CI could even reuse that same setting! [14:27:47] <_joe_> moritzm: I hope we will make our priority on day one to allow the gitlab CI to build and upload packages to our repos [14:28:02] <_joe_> because that would save so much time to us collectively [14:38:02] We build packages less extensively but we've had some success in leveraging fpm / fpm-cookery to make the builds a bit easier to manage [16:06:18] godog (or whoever), pontoon-logstash7-02.monitoring.eqiad1.wikimedia.cloud has a full disk drive and is erroring quite a bit [16:11:00] andrewbogott: I'll take a look [17:59:42] volans: what is up with puppet and the automation-framework project? there are two puppetmasters, one of them running Jessie, both broken... [17:59:46] should I try to fix that somehow? [18:04:29] * andrewbogott makes T266822 [18:47:33] andrewbogott: https://phabricator.wikimedia.org/T236582 [18:48:02] you should talk to chaomodus, but they are out for few days [19:40:08] ok! thanks [20:03:40] herron: FYI the Uncommitted DNS changes in Netbox seems related to logstash2032, is that you? [20:04:31] yeah, that can just be discarded [20:04:47] was running the cookbook to see uncommitted changes just now [20:05:00] still at Generating the DNS records from Netbox data. It will take a couple of minutes. [20:05:18] ok [20:16:30] volans: so why did mkvm for codfw end up causing uncommitted changes alert? the makevm prompts for a manual dns change there [20:18:05] herron: it shouldn't, you should get the automated dns changes show you the diff and get propagated also if they are not yet used because codfw is not yet migrated [20:18:15] and then tell you that a manual patch is still required for that record [20:18:45] did you get the diff for it during the makevm run? [20:18:58] yeah that's what it did, and at that point I hit ctrl-c. then a while later the uncommitted alert fired [20:19:27] yes did print the diff [20:19:47] if you ctrl+c-ed it and not confirmed you didn't push that change [20:19:52] that explains it [20:19:57] why did you had to ctrl+c it? [20:22:09] just to stop and regroup [20:23:02] but do you have to re-run it for real to create this logstash VM or not? [20:25:34] because re-running makevm will allocate a new ip, so you have to delete the existing one (and most likely makevm will re-allocate the same) from netbox before running makevm [20:26:37] ok, yeah will do it eventually. fwiw I did delete the allocation from netbox already [20:27:36] but then there should have been no discrepancy [20:27:40] and no icinga alert [20:28:23] unless you ctrl+c-ed after the dns commit but before the vm was created [20:29:16] feature request: trap SIGINT and SIGTERM and unwind everything :) [20:29:25] I hit ctrl-c at the point where it prints a diff and says to the effect of "a manual dns patch is needed or codfw, do that then type done" [20:29:31] (maybe SIGHUP too?) [20:33:42] it's not a diff there, that's after you confirmed the diff and pushed the commit [20:34:05] bblack: yes I already thought of that but is a mess in the general case based on which step it's in [20:34:47] volans: ok but how is this pushing a commit if dns is manually managed in codfw? [20:36:00] the generated data from netbox generates all teh records for all DCs, then that data is included into gdnsd manual files in the dns repo. All DCs but codfw have been migrated, codfw is the last one and will be done next week (or the one after at most). So the generated data for codfw is not yet used by gdnsd [20:36:20] but it's anyway generated and if there are pending changes that are not committed triggers the icinga alert [20:38:05] gotcha ok [20:43:14] thx for looking into it and explaining [20:45:10] np, sorry for the confusion during the transition time, it should all be over soon [20:47:13] np at all, was very nice building a vm with this in eqiad!