[04:11:41] <Krinkle>	 TIL `ls -l` has a column for showing the number of hard links an entry has. It's only shown if there are non-zero entries with >1 hard links, so pretty rare.
[04:12:06] <Krinkle>	 I noticed it in deploy1003:/srv/homedirs/mwmaint2002 when trying to figure out why I can't delete my own directory despite owning it.
[04:12:18] <Krinkle>	 (by doing `ll` which is `ls -halF` for me)
[04:17:01] <Krinkle>	 they al have different numbers of hardlinks to them, which seems odd. I can't seem to find any actual re-use of the inode numbers, though.
[04:19:51] <Krinkle>	 Hm.. maybe it just wasn't visible before. It's everywhere now. And it seems the minimum is not 1 but 2 for directories.
[04:22:41] <Krinkle>	 I thought  maybe it's counting the `.` self-references, but thought that couldn't be right as those aren't "real", and I doubt it's recursing/counting them on-demand. 
[04:23:15] <Krinkle>	 But turns out, at least indirectly through legacy/compat reasons, that is indeed exactly where the hardcoded offset of +1 comes from. https://unix.stackexchange.com/a/101536/37512
[04:23:39] <Krinkle>	 according to some guy on the internet, anyway
[08:02:03] <vgutierrez>	 is grafana loading for you guys?
[08:02:07] <vgutierrez>	 super slow here
[08:02:40] <vgutierrez>	 sigh, back to normal now
[08:04:09] <jynus>	 grafana is loading, but prometheus metrics are not for me
[08:06:05] <jynus>	 now they are back
[08:09:31] <vgutierrez>	 https://grafana.wikimedia.org/goto/30L46_EHg?orgId=1
[08:09:36] <vgutierrez>	 that matches what I was seeing
[08:09:49] <vgutierrez>	 (small spike of TCP errors on grafana1002)
[10:38:41] <vgutierrez>	 moritzm: mx[12]001.wm.o are long gone, right?
[10:43:54] <moritzm>	 yeah, they were replaced by separate MXes for inbound and outbound mail
[10:45:37] <moritzm>	 [12]1001 are gone even longer, the mx-in and mx-out servers replaced mx100[12] 
[10:45:47] <vgutierrez>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1164423
[10:46:03] <vgutierrez>	 no need to keep issuing a cert that can't be fetched by any server :D
[10:46:16] <fabfur>	 +1
[10:46:27] <moritzm>	 actully, 1002 were jiust test systems, I think used mx1001/2001 before they were split
[10:47:12] <moritzm>	 decom task was https://phabricator.wikimedia.org/T325409
[10:48:01] <vgutierrez>	 december 2022 :)
[10:49:12] <vgutierrez>	 rollback is pretty safe
[10:49:19] <vgutierrez>	 we don't delete certs
[11:46:16] <jynus>	 prometheus issues seems to be repeating
[11:46:33] <jynus>	 metrics not loading 50% of the time
[11:53:52] <p858snake|cloud>	 do the metrics support the claim >.>
[11:56:47] <jynus>	 only 50% of the time :-D