[09:25:23] <dhinus>	 is cloud-owner@lists.wikimedia.org an alias for another mailing list? I cannot find a list with that name in lists.wikimedia.org
[09:31:21] <volans>	 does it exits?
[09:38:03] <taavi>	 dhinus: in mailman, $LIST-owner always goes to moderators of $LIST
[09:38:16] <dhinus>	 ah that makes sense!
[09:38:45] <taavi>	 also, https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ has an auto-generated "To contact the list owners, use the following email address: cloud-owner@lists.wikimedia.org" bit
[09:39:17] <dhinus>	 thanks, completely missed it!
[09:39:28] <volans>	 righ, had forgot that automagic of lists
[09:40:07] <dhinus>	 has anyone replied to the email we received there ("PAWS: Extension broke")? I did a quick check and I cannot reproduce the problem
[09:41:44] <dhinus>	 I will reply to cloud-owner and cc the person who sent the request
[10:12:28] <dhinus>	 maybe I found what caused the increase in toolsdb disk space usage: https://phabricator.wikimedia.org/T427187#11971474
[13:13:51] <andrewbogott>	 dhinus: sorry, I followed up on IRC. That user was very complete, they asked on IRC and mailing list and also opened a phab ticket.
[13:15:41] <dhinus>	 LOL, I missed the other interactions as I was out... and saw the email this morning :)
[13:18:37] <dhinus>	 has anyone else encountered this issue? T427801
[13:18:38] <stashbot>	 T427801: [lima-kilo] SSL certificate errors after restarting the VM - https://phabricator.wikimedia.org/T427801
[13:26:04] <taavi>	 dhinus: is that the 'openldap data should really get persisted' issue?
[13:26:18] <dhinus>	 taavi: probably, because I just found that reloading users fixed it
[15:21:09] <bd808>	 IDP keeps freaking out when I try to log into https://prometheus-alerts.wmcloud.org/ with an "Application Not Authorized to Use CAS" error. Is this a me problem or an IDP problem?
[15:21:53] <taavi>	 bd808: 'application not authorized' would be a server-side problem
[15:24:50] <bd808>	 Am I using the wrong URL for alert manager?
[15:29:06] <andrewbogott>	 does https://alerts.wikimedia.org do the same?
[15:29:50] <andrewbogott>	 prometheus-alerts gives me the same CAS error and has for a few days
[15:30:05] <taavi>	 prometheus-alerts should work
[15:30:56] <andrewbogott>	 should, or does for you?
[15:31:12] <taavi>	 should
[15:31:22] <bd808>	 nope. I can use https://idp.wmcloud.org/logout, but then if I visit https://idp.wmcloud.org/login I end up with the same "Application Not Authorized to Use CAS" error
[15:31:58] <bd808>	 so either CAS is hosed, or logout keeps my identity and loops back to a data error problem of some sort related to me.
[15:33:33] <andrewbogott>	 same for me, so I vote 'CAS is hosed'
[15:33:38] <andrewbogott>	 although https://alerts.wikimedia.org still works for me for some reason
[15:33:51] <bd808>	 andrewbogott: different CAS services
[15:34:25] <andrewbogott>	 oh yes, I was only reading the right-half of the urls
[15:35:46] <bd808>	 T427826
[15:35:47] <stashbot>	 T427826: "Application Not Authorized to Use CAS" error when attempting to authenticate to IDP - https://phabricator.wikimedia.org/T427826
[15:36:09] <taavi>	 2026-06-01 15:36:00,072 ERROR [org.apereo.cas.ticket.registry.MemcachedTicketRegistry] - <Timeout waiting for value: waited 2,500 ms. Node status: Connection Status { localhost/127.0.0.1:11000 active: false, authed: true, last read: 3,014,143,826 ms ago }
[15:36:29] <taavi>	 ^ from cloudinfra-idp logs
[15:42:28] <taavi>	 bd808: try now?
[15:42:47] <andrewbogott>	 works for me now
[15:44:28] <bd808>	 taavi: fixed for me
[15:45:09] <andrewbogott>	 thx taavi 
[15:45:30] <taavi>	 this is very unsatisfying but somehow the integration of unattended-upgrades and our custom puppet-shipped memcached unit file caused systemd to, uh, not start memcached again after it was upgraded?
[15:46:01] <taavi>	 like, systemd thought the service was running but with no processes
[15:46:06] <bd808>	 "neat" :/
[15:52:55] <andrewbogott>	 good job systemd
[16:20:50] <volans>	 sigh! (/me just cathcing up after meeting)
[16:59:36] <komla>	 Anyone with the bandwidth to look at this for me: 
[16:59:36] <komla>	  https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294864
[16:59:36] <komla>	 A systemd timer update. Thanks!