[11:19:19] I'd appreciate feedback on the comments/rationale of https://gerrit.wikimedia.org/r/c/operations/puppet/+/564959 btw (no partman recipe reading required) [11:19:43] <_joe_> godog: you said partman, we're running away anyways [11:21:18] lol, trying to reestablish partman name one recipe at a time [11:21:41] we're running the risk of actually understanding what's going on in netboot.cfg with the standard recipes [11:24:35] <_joe_> lies! [11:37:34] hhaha you'll see [15:20:17] moritzm, reprepro is messing with me.. maybe you can help me figuring out why [15:20:29] root@install1002:~# reprepro -C main include buster-wikimedia ~vgutierrez/libvmod-tbf/libvmod-tbf_2.0.91-2wm_amd64.changes [15:20:29] File "pool/main/libv/libvmod-tbf/libvmod-tbf_2.0.91.orig.tar.gz" is already registered with different checksums! [15:20:40] looking [15:20:49] so I've fetched the source package on boron with apt-get source [15:21:01] and ran a quick check with diff -r against master on the repo [15:21:07] vgutierrez@boron:~$ diff -rq libvmod-tbf/ t/src/libvmod-tbf-2.0.91/ [15:21:07] Only in libvmod-tbf/: .git [15:21:15] so I'm a little bit puzzled [15:21:47] the changes introduced on 2.0.91-2wm only affect the debian branch, https://gerrit.wikimedia.org/r/c/operations/software/varnish/libvmod-tbf/+/565550 [15:21:58] but obviously I'm missing something [15:25:21] hmmh, the sha1 sums in fact differ, the one currently on install1002 is 28164fc5ff5e22c49581062709233f4ac737c731 and the one in your home is 5c8449cd66841223d8260704142b6e038357860c [15:25:43] yep [15:26:03] but diff seems to think that the content is the same [15:26:13] let me checksum every single file... [15:29:34] so.. a md5sum of every file shows the same md5... [15:29:35] WTF [15:29:49] I think due to mtimes the overall archive checksum will still differ if newly generated, can we reconstruct how the tarball from the original tarball was constructed, is that an external project or a WMF local software? [15:31:47] external one with local mirror @ operations/software/varnish/libvmod-tbf [15:32:41] moritzm: considering that's the same version and the orig is already there I guess that if I skip uploading the orig source it should work, right? [15:33:44] yeah, that should work fine, did you pass --buildopts -sa to the build? then you can simply omit that (or simply remove the references to the orig tarball from the changes file, that also works fine) [15:34:32] yep.. -sa is there [15:39:35] ok, then simply drop it and rebuild and just edit the changes file, you only need to remove the three lines with the md5, sha1 and sha256 checksums [15:39:49] hmm so.. without -sa the orig.src is not there anymore [15:39:55] but reprepro seems to complain anyways [15:41:04] should I clean something before retrying? [15:41:45] with the current version of ~vgutierrez/libvmod-tbf/libvmod-tbf_2.0.91-2wm_amd64.changes from 15:37? [15:41:50] really? [15:42:06] that's odd [15:42:16] moritzm: https://phabricator.wikimedia.org/P10236 [15:42:21] that's the whole output [15:42:54] so I'm wondering if the previous one got cached somehow [15:48:43] ah, reprepro actually parses the .dsc file! and the dsc actually references the check sums for the orig.tar.gz which has been regenerated (5c8449cd66841223d8260704142b6e038357860c) [15:49:53] :_) [15:50:19] hmm can I get somehow the original orig.tar.gz to boron to use that in the building process? [15:51:29] yeah, that's what we could do: [15:52:50] sync the source tree to boron and copy the tarball from install1002:/srv/wikimedia/pool/main/libv/libvmod-tbf/ to the directory level below the debian root dir and then build with "DIST=buster-wikimedia pdebuild", then it will use the local tarball [15:53:57] <3 thx [15:55:27] actually I already got that in boron.. I've fetched it using apt-get source [15:55:34] ack [15:56:20] if we foresee that we'll need to rebuild vmod-tbf again (or if it's going away anyway soon because of ATS), then we could also sort this out more cleanly, but if this is just a one time rebuild to move to buster, then we can just workaround with a local build... [16:02:48] moritzm: thanks :) that solved my L8 issue [16:03:07] cool :-) [16:03:35] I don't think is going away soon [16:03:43] varnish-fe is going to be there for a while [16:10:42] is there ongoing upstream development for vmod-tbf? is so, the problem will also vanish automatically whenever we upgrade to the next release [16:16:15] probably is going to be updated if we decide to upgrade to varnish 6 [16:16:32] if varnish-fe is going to be with us for a while that's going to be necessary at some point [16:19:31] ack [16:58:45] going to reboot pfw3-eqiad very shortly for sofware upgrade, should take about 20min. There might be some alerting noise but I downtimed the firewalls [19:38:20] apergos: , yt? [19:38:28] did the dumps nfs hosts change? [19:38:29] https://phabricator.wikimedia.org/T243328 [19:38:40] we've got e.g. dumps-labstore1006.wikimedia.org [19:40:13] i know about the puppet compiler needing to sync facts for new hosts.. but in this case it already knew one and then forgot it again .. huhmm [19:42:23] ottomata: there's been work done on the related manifests but you want to check in wikimedia-cloud-admin [19:42:45] I'm out of the loop on that (also I'm pretty much on goof-off time here) [19:54:32] I'm going to reboot mr1-esams for software upgrade, icinga is downtimed, but never 100% safe of some icinga noise, should take 20min or so [19:54:45] !log puppet-diffs syncing puppet facts from local puppet repo [19:54:45] mutante: Not expecting to hear !log here [19:54:49] thanks XioNoX [20:04:46] wikimedia-cloud-admin .... [20:17:13] ottomata? [20:18:39] ottomata: b storm is on the case, see the ticket [21:33:00] Ah great, thank youO! [21:33:40] bstorm_: i saw that too (was in a meeting) let me know if you want me to fix. I think I can just change the paths to the real ones (instead of just /dumps) right? [21:34:24] https://gerrit.wikimedia.org/r/c/operations/puppet/+/566365 <-- in the commit msg there's a note on using the data [21:34:28] But that should fix the mount [21:38:41] hm bstorm_ [21:38:42] hm [21:38:44] 2 qs [21:39:03] wouldn't that mean the new path on the client be /mnt/data/srv/dumps/xmldatadumps/public [21:39:04] ? [21:39:12] no [21:39:21] nfsv4 exports in a funny way [21:39:33] In this case, the export is fsid=0 so it's / [21:39:50] This is already all over the cloud. I forgot the analytics mounts (sorry) [21:39:53] somehow nfs / == /srv/dumps ? [21:40:35] At this point nfs / = /mnt/data/srv/dumps/xmldatadumps/public [21:40:38] sorry [21:40:44] I meant: [21:40:51] ok, fortunetly i think most people use a historical symlink we maintain at /mnt/data, i can point that at the new path [21:41:02] nfs / = /srv/dumps/xmldatadumps/public [21:41:09] oh right [21:41:10] so [21:41:29] https://gerrit.wikimedia.org/r/c/operations/puppet/+/566365/1/modules/statistics/manifests/dataset_mount.pp [21:41:30] This was a security issue [21:41:33] you changed it to mount at / [21:41:34] right? [21:41:38] OH i see [21:41:40] Yes [21:42:04] so the mount point on the client will be the public/ dir [21:42:13] Yes [21:42:21] https://www.irccloud.com/pastebin/5eopCtgp/hm [21:42:26] ack didn't mean to pastebin [21:42:33] We can change the link actually in that patch... [21:42:39] right.... [21:42:47] Create dirs down and make it match. One sec [21:42:51] i'm actually not familiar with what /dumps was before [21:42:55] was it public/ [21:42:56] ? [21:43:04] or just /srv/dumps [21:43:04] ? [21:43:18] ah i think i see [21:43:20] ok ty [21:43:24] :) ok [21:43:37] It was exported way too high up on the filesystem [21:43:50] Now it's better [21:54:14] ottomata: there is a link one level above that dir that won't be there unless we also add it (looks like 'incr -> public/other/incr/'). [21:54:25] I don't know if that's in use or not [21:55:25] hm, i see, i don't know either [21:55:30] i'm not familiar with it [21:56:05] bstorm_: let's skip it and wait til someone complains :p [21:56:09] fixing mount is better i think [21:56:18] I'm good with that [21:56:39] Running compiler to check my changes to the link bit [22:04:06] I'm imagining I'm going to have to clean up that old link manually...we'll see in a moment :) [22:04:20] nope [22:04:27] It changed "link" to "directory" :) [22:05:32] ottomata: `ls /mnt/data/xmldatadumps/public/` now produces a suitably large amount of stuff on the terminal on notebook1003 [22:05:50] nice perfect thank you [22:18:23] the incr link we presumed some projects or scripts might have relied on it [22:18:31] but I don't know it to be true [22:22:41] i know generally that it is used by dumps users but no idea about internal users of them