[08:54:09] hey folks, as FYI I am running tpctop on wikikube-worker{1016,1070} to see if I can narrow down the process using the bw (to confirm it is only thumbor) [09:01:31] elukey: so graphs still look bad even after the chunking change in thumbor? (I did not check yet) [09:03:07] yeah [09:03:23] :/ [09:04:04] I'll do my reporting chores and would deploy the thumbor-affinity changes after - when you're done taking your measurements [09:05:32] tragedy [09:05:53] at least it seems like these machines _have_ 10G interfaces [09:06:44] yeah, DC-Ops will set those up shortly https://phabricator.wikimedia.org/T440696 [09:07:08] we'll "just" have to to reconfiguration of the hosts and switches/BGP :/ [09:11:33] so far I am mainly watching mcrouter-main-m7xp5 but of course now it is flat zero [09:11:51] so if you are already ready for the affinity I'll stop everything [09:12:03] it was just to have some more info but probably we don't need it [09:13:23] no worries, I have a bunch of things to take care of. Deploying the affinity can wait some more hours [10:07:22] bjensen: if you have a minute I've prepared https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1354388 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354390 to scrape pod resource metrics from the kube-scheduler [10:07:50] I'd like to use those to improve https://grafana-rw.wikimedia.org/d/pz5A-vASz/kubernetes-resources for capex estimations [10:08:02] for sure, will take a look momentarily [10:08:06] <3 [10:08:37] running pcc [11:46:24] jayme: I stopped the tcptop experiments, nothing really popped up so far, you are free to go [11:53:46] okidoke, thanks [12:32:51] I did a big overhaul of the https://grafana-rw.wikimedia.org/d/pz5A-vASz/kubernetes-resources dashboard. It is now "more correct" (taking init containers properly into account and accounting for gVisor overhead) and also faster since it now uses accumulated resource metrics from the k8s scheduler rather than querying all containers. It also now excludes all control planes and containers on control planes from all calculations [13:02:39] PriorityClass is fascinating [13:02:49] "This means that the range of values for a PriorityClass object is from -2147483648 to 1000000000 inclusive." [13:03:47] i dunno why you'd define a configuration like this and allow negatives, that just seems to be opening the door to a lot of unnecessary confusion [13:05:05] mmmmm, nevermind, i guess if 0 is the default, you might want negative values for batch-whatever that you'd like to not trample on more important workloads [13:05:13] still looks funny [15:42:44] jayme: IIUC you already deployed the affinity rule in eqiad and codfw right? [15:42:54] yesno [15:43:16] ahokok I'll wait [15:43:18] <3 [15:43:22] I deployed stuff, but theres a bug in the thumbor chart [15:43:36] I didn't see improvements in the mcrouter metrics and I got sad [15:45:29] elukey: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1354487 if you have a minute [15:46:13] I'm not sure why CI did not catch that..we probably ignore superfluous keys [16:06:20] sorry arrived late :) [16:13:28] elukey: np. deployed now to both DCs [16:16:43] first datapoints are good but we'll need to wait some hours, fingers crossed [16:18:18] yeah...we'll see tomorrow I suppose [16:28:22] jayme: it already looks extremely good though [16:28:35] right?! [16:28:38] https://logstash.wikimedia.org/goto/0caaf41d89b790a33992f4337046c4cd [16:28:51] https://grafana-rw.wikimedia.org/goto/sqsr6g?orgId=default [16:29:11] yeah pretty sure we are good now [16:29:27] finally :D