[13:51:41] hello k8s f5s, I've come here to discuss https://phabricator.wikimedia.org/T428573#12010077 [13:53:42] tl;dr it's *really* nice for the DBAs to have reverse DNS on k8s pod IPs that are accessing the database, we have this working via zone delegation to coredns in prod https://phabricator.wikimedia.org/T344171 [13:53:52] but that only works for pods that are part of an Endpoint [13:54:38] I think the easiest option is add headless services for any mediawiki job, or for anything that touches the DBs from prod (but that's more work for serviceops) [14:06:32] Hello. I'm also going to drop this here: https://phabricator.wikimedia.org/T428895 - It's just an FYI, really, but we're going to go ahead and try out the TopoLVM CSI driver. https://github.com/topolvm/topolvm [14:06:42] in dse-k8s obvs. [14:07:00] neat [14:10:18] I've only used https://github.com/rancher/local-path-provisioner which is much simpler [14:30:39] ^^ on that tip, does anyone know what happens to emptyDir volumes when the worker is rebooted? Would k8s give up on the pod hosted on the worker and schedule it somewhere else, losing data? [14:37:50] pod killed -> data gone [14:37:55] for any reason [14:38:03] (containers can crash and get restarted, though) [14:38:16] https://kubernetes.io/docs/concepts/storage/volumes/#emptydir [14:43:00] Thanks, I was looking over that earlier, just making sure. And if you wanted emptyDir data to survive a reboot (putting aside the fact that you're abusing the concept of emptyDir), I'm guessing you would have to detune your health check? [14:44:24] for pods of a Deployment or a StatefulSet it will certainly not survive the reboot [14:44:41] you cannot do what you want to do with emptyDir [14:46:10] in fact I'm not even sure if any pods survive reboots [14:46:15] probably not [14:47:03] ah okay, daemonset pods do [14:47:23] but I still wouldn't count on their emptyDir data surviving :) [14:51:36] ACK, but with the local storage provisioner or TopoLVM, the pod would die during a reboot. But it would be rescheduled on the same worker with the volume once it was available? [14:52:37] yes, at CSI provisioning time, they create PVCs that are bound to the host, and that are not just in local filesystem scratch space [14:53:11] one of the benefits of topolvm is it influences scheduling -- local-path-provisioner just makes a volume wherever the default scheduler lands your pod [14:56:14] I'm thinking of the WDQS use case where we are running an initContainer that copies data from S3 on pod deployment. It seems like that process would still happen even if we used TopoLVM, since the pod would be killed [14:56:56] I guess we could make the initContainer check for existing data [14:56:59] it would have to be idempotent, yes [14:57:16] just make it `rclone sync` ;) [15:00:26] Thanks, this helps me understand some of the benefits