[15:56:58] Amir1: I'm a bit surprised apcu inserts hasn't gone down after your rdbms backport. [15:57:00] https://grafana.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s?from=now-24h&to=now&timezone=utc&var-dc=000000018&var-namespace=mw-web&var-release=main&var-container_name=$__all&var-site=arwiki&var-kubernetes_pod_name=All&viewPanel=panel-60 [15:57:09] (click legnd to show only inserts) [15:57:19] started a few hours before your backport [15:57:37] spikes are normal after a deploy/restart but it's not levelling out [15:57:42] yeah, I tried to see what's going on. Nothing obvious stood out [15:58:12] rest of dash suggests rollout spike has finished and levelled out, e.g. memory size and entries count [15:58:26] misses did go down [15:58:35] so that might be due to your change [15:58:53] but unlikely because you didn't change storage ttl [15:59:01] only fresh ttl decision to inform whether to write [15:59:45] ah also, I think some scraping going on [18:37:21] attn all kubernetes service deployers: I just updated the global default mesh.image_version, which affects all services -- I'll deploy ~everything over the next couple of days, but if you see that in your helmfile diffs before I get to it, it's thoroughly tested and safe to apply [23:06:23] what does it mean to "log [something] in the server admin log" [23:06:42] is that just the bang log command in any channel [23:22:14] lizimedia: any channel with stashbot in it yeah, most commonly #wikimedia-operations [23:22:19] and then it automatically goes to https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:22:24] or, searchably, https://sal.toolforge.org/ [23:23:28] I initially thought you just edited that wiki page with whatever command you ran [23:23:53] you can! It's Literally Just A Wiki etc etc. but it's rare, using !log is both easier and good for visibility [23:26:20] https://wikitech.wikimedia.org/wiki/Debian_packaging#Rebuilding_a_package wikitech needs {{When}} for entries like these [23:26:56] when the bot isn't available or working (which happens on rare occasions), a good samaritan may go back through their channel logs, find the entries that are missing, and add them to the wiki manually. very rarely though. [23:36:30] please don't go manually edit the wiki page, or otherwise https://sal.toolforge.org/ (which provides a nice searchable and linkable index) will get out of sync [23:37:09] also !log doesn't work the same in all channels, best to stick to -operations