[06:08:05] ryankemper: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308355 this has been sitting waiting to be merged for 2 days [06:08:07] I am going to revert it [07:30:28] marostegui: grr, could have sworn i had puppet-merged. thanks for cleaning that up for me [07:30:44] ryankemper: no worries! it happens :) [08:36:02] hey on-callers [08:36:38] I am going to turn request logging to debug on kafka-main1008 to investigate a weird issue, and a possible root cause for mirror maker not working between jumbo and main-eqiad [08:37:07] thanks elukey [08:43:13] ack [10:51:02] jelto: are you taking care of deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/1317847 or should I? [10:51:48] I'd be happy if you deploy the change, but I can also do that [10:52:59] will do [10:54:08] thank you! [11:21:40] hello! I'm going to depool ulsfo for a router upgrade and deploying a new feature [11:22:17] Raine, federico3 ^ [11:23:11] ack [13:06:50] all done, repooling ulsfo [15:13:02] Some news in the mysterious sendmail bug for confirmation emails from mw-on-k8s - there is now a consistent repro and two volunteers able to help test. https://phabricator.wikimedia.org/T383047 [16:18:48] ooh great, thanks Krinkle [17:12:31] a significant fraction of emails seem to be failing [17:12:35] possibly all of them [17:15:14] (Works for me in a quick test, so not all of them. But the error rate is very high.) [17:16:48] or maybe it's a logging error - I got the error, but there is an error for that account creation request in the logs [17:17:45] is anyone using the mw infrastructure window? [17:45:01] ^ most often a question for swfrench-wmf [17:45:20] nothing planned on my end :) [18:20:07] Is there a secondary way to find k8s app logs? Specifically `kubectl logs ...` shows me logs from today, but `App Logs (Kubernetes)` in logstash shows no logs today. Suspeting i'm doing something wrong. [18:21:58] it's roughly https://logstash.wikimedia.org/goto/113a34a8216fb45d31feff59cf0bcb8c vs `kube_env cirrus-streaming-updater eqiad; kubectl logs flink-app-consumer-cloudelastic-backfill-6bcf64c94b-nl85q flink-main-container` [18:24:25] tgr_: swfrench-wmf and I are looking into it [19:36:27] swfrench-wmf: RE T432983, I'm going through the php-apcu changelog. It's had more than the usual amount of change in a year, some interesting one that I do think will affect us. [19:36:28] T432983: [PHP 8.5] Create Debian packages for PHP 8.5 - https://phabricator.wikimedia.org/T432983 [19:36:48] putting it together now, but curious if you had a look already [19:54:06] Krinkle: I'll be taking point for the 8.5 upgrade, with Scott to lend his brain when mine falls short :D I haven't looked through the changelist yet, would love to get your thoughts. [20:11:29] Perfect. I'll post it shortly [20:11:46] Raine: one thing I'm looking at is these stats: https://grafana.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s [20:12:37] I have some doubt about whether they're correct. [20:14:21] Krinkle: you mean rps? [20:14:42] or APCU specifically? [20:14:46] APCU yeah [20:15:00] (sorry still picking up context) [20:15:04] The ops/sec nubmers are from our instrument: https://gerrit.wikimedia.org/g/operations/docker-images/production-images/+/27a63aaa4b56462e0c1c4a67be70fe8ec6f10c18/images/php/common/fpm/monitoring/lib.php [20:15:26] but afaik most of the numbers we report as counters there are actually gauges [20:15:41] eh oops :D [20:15:49] ok, I'll have a look and fix the graphs accordingly [20:16:06] (should we be fixing the graphs, or making the code produce counters?) [20:16:31] I don't thikn we can get counters from the native code. [20:16:36] right, because php? [20:17:04] well, unlike most php, this is a long running process, it's the parent php-fpm process where the shared memory is continuous [20:17:13] but the native code gives us gauges [20:17:21] oh, right [20:17:22] hm [20:17:27] the grpahs can't be fixed I think [20:17:32] we need to change our instrumentation, if I'm right [20:17:49] the middleware I linked above that takes them from native to prometheus [20:18:28] I'm trying to pull out some numbers from the graph that is more meaningful to me. rather than sum() across pods, I'm creating some per-pod numbers so that I can relate it to a given pod and its req/s to see if it makes sense or not. [20:20:17] yeah, that makes sense [20:27:12] are they really broken? [20:27:29] * Raine needs to look at the native code [20:27:33] OK, so I've added a por pod graphs, plotting the p10/p50/p90 pod [20:28:43] the numbers are definetively considered gauges internally, but depending on the level of indirection we can and should expose them as counters to prometheus, given that prometheus counters are meant to keep going up durig hte lifetime of a pod. [20:29:09] so num_hits and num_misses/total_gets is probably fine, so long as there is no additional layer reporting each subtotal as an increment [20:29:21] entries though, afaik, can go both up and down [20:29:48] my worry was that we're reporting each subtotal as an increment on each scrape, but that woudl only happen if we had something like statsd in the middle [20:30:00] statsd-exporter I mean, so I think most are fine indeed. [20:30:35] yeah, this goes directly to prometheus [20:30:57] total_gets: https://grafana-rw.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s?orgId=1&from=now-6h&to=now&timezone=utc&var-dc=000000017&var-namespace=mw-web&var-release=main&var-container_name=$__all&var-site=arwiki&var-kubernetes_pod_name=All&refresh=1m&forceLogin=true&showCategory=Panel%20options&viewPanel=panel-349 [20:31:11] given 2-4K req/s, having 4K op/s on apcu seems realistic [20:31:41] idem for inserts, looks probable (800/s) [20:31:45] entries: https://grafana-rw.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s?orgId=1&from=now-6h&to=now&timezone=utc&var-dc=000000017&var-namespace=mw-web&var-release=main&var-container_name=$__all&var-site=arwiki&var-kubernetes_pod_name=All&refresh=1m&forceLogin=true&showCategory=Panel%20options&viewPanel=panel-351 [20:31:47] that looks funky [20:35:21] "usage (total)" is measuring memory usage but "usage (per pod)" is measuring memory free. That one makes sense to report free, but I'm fixing the label [20:36:12] thank you [20:36:45] yeah, so entries looks broken. it doesn't align well with deployments as you'd expect (by eyebaling other metrics and their natural curve) [20:37:03] and i'm not sure about having just 4K entries total. [20:37:29] presumably that's being interpreted as a counter, and so we're plotting the relative change in entries count. which is initially about 4K per second and then stabilises [20:37:38] but that just duplicates inserts then (sans overwrites) [20:38:01] so we don't know how many entries there are which is presumably what you'd want to know from that metric [20:38:17] and when it goes down, prometheus presumably gets confused [20:39:23] I can get some emperical confirmation via mwdebug shell and see it go up and down the way I expect [20:48:10] * Krinkle uses T405688 to launch mw-exp-shell and then uses `echo file_get_contents("http://localhost:9181/metrics");` from php -a, given no curl there. [20:52:35] okay, confirmed, it can go down. [20:52:58] I created /w/krinkle.php on mw-experimental and had it apcu_store something, then scrape metrics, then apcu_delete that key and scrape again. [21:11:09] Krinkle: that makes sense I suppose [21:11:24] so, it just needs to be used as a gauge in grafana [21:11:36] (ideally also marked as a gauge in prometheus, but that's details) [21:13:06] Oh I see what you mean, the data hasn't been corrupted [21:13:28] I was going to rename it in the middleware because it's currently grouped under a counter. [21:13:38] you can't change the type of a single label combo I think. [21:14:32] yeah it'd need to be the whole metric [21:14:36] not just one label [21:14:47] and yes, the data is stored as is, just graphed wrong [21:14:57] it's pretty late over here, but I can fix up the grafana quieries tomorrow [21:15:25] it's currently under apcu_cache_ops{type="hits"|"misses"|"inserts"|entries"} [21:15:44] https://gerrit.wikimedia.org/r/1318289 [21:16:26] but yeah, so long as we are careful never to rate(apcu_cache_ops) by (type) without excluding type=entries, than that would work [21:16:30] it's def a trap though [21:18:56] I've edited the dash to remove entries from the ops, and replotted the other one as a gauge [21:21:19] (saved)