[10:10:55] thanks legoktm! [12:45:19] what info is stored on the redis cluster running on the MediaWiki memcached servers? and does it have to be persistent when replacing servers? [12:48:08] puppet labels those as "memcached+redis sessions" but on mw side they are configured as "redis locks" [13:37:18] Majavah: there is more than one redis cluster [13:37:20] which one? [13:38:18] Krinkle: the one that mediawiki config services file labels as "redis_lock" [13:38:34] for context I'm trying to replace deployment-memc[04-05] which are running Jessie with Buster variants [13:39:08] we have an "rdb*" redis cluster for locks used by FileBackend, and there is a redis cluster co-hosted on the memc* machines used by MainStash. [13:39:20] I dont know how it is set up in beta, maybe the same one is used for both? [13:40:22] I'm not sure at all [13:40:29] where is the MainStash cluster configured? [13:41:03] 'wgMainStash' => [ 'default' => 'redis_local', [13:41:33] connects via nutcracker [13:41:38] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/7015b203029182e5be5c3f3197788b8549b14487/wmf-config/redis.php#21 [13:41:46] backends declared in puppet [13:42:14] the data in main stash is meant to persist strongly, but loss is fine if rare (couple times a year?) [13:42:18] and for beta it doesn't matter [13:42:50] it has some visible effects on user, but nothing essential [13:44:25] via hiera, so that's "profile::mediawiki::mcrouter_wancache::shards"? [14:42:42] Majavah: no, mcrouter is the local proxy on mw servers for memcached. For redis it is nutcracker [14:44:35] But none of this matters for beta cluster. As long as you find all references to hostname and IPs in wmf-config, puppet, and horizon/hiera and update them it is okay [14:45:09] Nothing in redis or memc in beta is primary data that needs to be moved to a new server and may be lost or down for a little while [14:46:24] Can create replacements. Switch config. Shutdown old. And after a week delete the shutdown old [14:46:39] That's what I'd do anyway :) [14:58:17] Krinkle: okay, thanks! next problem is that I can't find where deployment-mediawiki-07 gets its nutcracker hosts, looked in both horizon and ops/puppet but the only references for those IPs are in the devtools project [15:23:24] ah, found them, not sure how I didn't see them in horizon project puppet hiera [16:33:41] https://gerrit.wikimedia.org/r/669436 is a beta-only ops/puppet patch, merge would be appreciated, thanks [16:34:40] ack, LGTM. though I'm not an SRE so that may have to wait until Monday. :) [16:35:37] Krinkle: posted here to get SRE attention :P [16:37:29] technically I could cherry pick it for now and let SREs merge it tomorrow