[10:09:17] o/ I added a new repo to apt-staging2001 and I'm trying to fetch it from apt1002, but reprepro checkupdate is not finding it [10:09:28] I suspect that's because text-lb is caching the results from apt-staging.wikimedia.org [10:19:13] hmm maybe text-lb is a red herring, but I do see cached results if I curl apt-staging.wikimedia.org from apt1002 [10:19:35] where is that cache and how can I invalidate it? [10:37:31] dhinus: I would guess that's the CDN cache, the response headers would confirm that. the way to do one-off purges is with the purgeList mediawiki script https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging#One-off_purge [10:39:40] taavi: thanks! [10:40:03] <_joe_> dhinus: I never had this issue FWIW [10:40:46] <_joe_> to confirm the problem is caching, just add a random query string [10:41:20] <_joe_> dhinus: have you checked that the script that imports the repo in apt-staging is running and did import your repo correctly? [10:41:23] _joe_: yes, with the query string it fetches the updated version [10:41:47] the package is there in apt-staging [10:41:57] <_joe_> ok then purge the url as taavi suggested [10:42:06] <_joe_> which one is the url that is cached? [10:42:13] I think it's https://apt-staging.wikimedia.org/wikimedia-staging/dists/bookworm-wikimedia/main/binary-amd64/Packages.gz [10:42:21] <_joe_> ack [10:42:22] but I'm not sure if that's the one reprepro is using [10:42:27] but that one is definitely cached [10:42:55] when I run "checkupdate" I don't see anything reaching the nginx access log in apt-staging [10:43:12] <_joe_> so it's cached at the CDN [10:43:16] <_joe_> yeah [10:43:35] yes, the additional complexity is that "checkupdate" might be fetching another URL which is also cached [10:43:46] <_joe_> purge that url and then maybe it's a good idea to add some caching headers to responses fom apt-staging to keep the cache at say 5 minutes [10:44:02] yeah I was thinking of something like that, I can create a phab task [10:44:23] yeah, I was just about to say that as well [10:46:30] URL purged, but checkupdate is still not happy. I think I saw another URL in the access logs, let me check [10:47:26] "/wikimedia-staging/dists/bookworm-wikimedia/InRelease", maybe that's the one [10:49:56] yes! that one did it [10:50:27] I'm creating the phab task, which tags should I add? [11:02:29] T402284 [11:02:29] T402284: apt-staging: add headers to prevent CDN caching - https://phabricator.wikimedia.org/T402284 [14:56:12] pcc seems stuck (been waiting >20 minutes for a run) -- is there something I can/should do about that? [16:27:43] jasmine_: hi! I'm having issues trying to import your gpg key to add a new file to pwstore (can't find it on https://keys.openpgp.org/), did you publish it? [16:33:16] dcaro: hi! i believe so, https://keys.openpgp.org/search?q=jkilani%40wikimedia.org perhaps? [16:33:36] hmm, that fingerprint is different than the one in pwstore [16:33:39] let me try though [16:35:14] I was able to import the key ok, but `pws ed` still tries to look up one with id 7554A25E95C29AEF92F4BC2150356D98462619A1 [16:45:54] jasmine_: ^ did you use a different key maybe? [16:47:46] btw. welcome :) [16:50:47] ah yeah I see, that's my expired key, let me export my non expired one [16:50:48] also ty! [16:54:52] found the key for import under `keys/` actually xd [17:03:08] oh whoops, my bad yeah that's actually the right non-expired one [17:03:12] and awesome, glad it worked! [19:32:15] I think the "@memorysize" core fact we use in some puppet erb templates stopped working in puppet on trixie. Tried trixie image on cloud VPS and hunted puppet error down to "failed to parse template". https://www.puppet.com/docs/puppet/7/core_facts.html says it's "legacy fact" that is "hidden by default in facter output" [19:32:57] heads-up that I'm finishing off the rolling restart of search eqiad. I've depooled eqiad and suppressed all cirrussearch1* alerts for the next 2 hrs, but there may be some alerts [19:33:54] we use the same Puppet 7 agent on trixie as we do on Bullseye and Bookworm [19:34:40] the Puppet 8 agent (which no longer supports legacy facts) will only be used once the fact migration is completed [19:35:54] mutante: IIRC, legacy still works (and I think we use some of those facts in other places). it's just that "hidden by default" [19:35:55] hmm, interesting. thank you [19:36:13] something else must be going on then.. ack [19:36:31] you could use $facts['memory']['system']['total_bytes'] in theory (in bytes though) [19:36:33] it's the only variable on that template line it fails to parse though.. hm..ok [19:36:37] this is what we do on the cp nodes [19:36:44] interesting [19:36:52] *nod*, thanks [19:47:02] annnnd...the rolling restart is done [20:03:25] the full thing is like: <%= (Float(@memorysize.split[0]) * 0.75).round %> maybe it has to do with the Float or round part then. [20:04:26] is there anyone here with strong `megacli`-fu? swift uses a "JBOD" made up of single disk raid0s, and I can't suss out how you map the kernel's word-view with that of the raid controller [20:05:08] like... given a device `/dev/sdg`, how does one get to a slot on the raid controller? [20:05:36] s/swift/ms/ fwiw [20:05:56] T402346 for context [20:05:57] T402346: hw troubleshooting: disk (sdg) errors on ms-be1071 - https://phabricator.wikimedia.org/T402346 [20:07:55] I know this is possible, Empero.r must be doing it on the regular, but it is not obvious to me [20:55:43] urandom: I don't know anything about megacli but, if you do decide to take the drive out of the swift rings, that I at least used to know how to do [20:56:20] cdanis: thanks; I found this https://wikitech.wikimedia.org/wiki/Swift/How_To#Remove_(fail_out)_a_drive_from_a_ring [20:56:37] I guess it depends on how long the failure will last? [20:56:50] basically, yeah