[06:21:35] cdanis: I just saw https://gerrit.wikimedia.org/r/c/operations/puppet/+/633229, thanks <3 [07:39:40] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) Thanks for trying that out. I ran the same command for much longer with higher concurrency. For Main_Page, I saw no difference. Then... [08:00:30] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I've checked with curl and it does get cached by both hosts: ` X-Cache: cp3060 miss, cp3052 hit/1000011 X-Cache-Status: hit-front... [08:27:38] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) >>! In T264398#6535510, @Gilles wrote: > Thanks for trying that out. I ran the same command for much longer with higher concurrency. For... [08:53:49] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I tried a bunch of different benchmark settings before I ended up using those, some of which definitely had too much concurrency and... [09:04:27] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) If it was indeed the last benchmark run, what you're saying is that cp3052 favored the benchmark requests over others, while cp3054 r... [09:47:23] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) >>! In T264398#6535643, @Gilles wrote: > I've checked with curl and it does get cached by both hosts: [...] > Oddly, cp3054 has Content-... [10:22:57] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) It was: ` gilles@cp3052:/home/ema$ curl -I -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.org" http://localhost/wiki/Barack_Ob... [10:24:58] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I guess the difference is that I did a HEAD request. [10:42:32] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Update 2030.wikimedia.org redirect to new URI - https://phabricator.wikimedia.org/T264797 (10JMeybohm) p:05Triage→03Medium a:03Dzahn Assigned to @Dzahn, since it looks as if a additional review of yours has been requested. [12:40:04] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) >>! In T264398#6536169, @Gilles wrote: > I guess the difference is that I did a HEAD request. It's still reproducible right now. OK, I'... [13:50:39] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Update 2030.wikimedia.org redirect to new URI - https://phabricator.wikimedia.org/T264797 (10Dzahn) a:05Dzahn→03None [14:00:02] ema: <3 there's some more fun on the related bug, in retrospect I feel silly it took so long to realize the random page endpoint was most of the trouble [15:27:49] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) >>! In T264074#6507717, @elukey wrote: > Most of the usage seems to be VUT related, especially for `fxstatat64` (no idea where it is used). You are indeed correct. The n... [15:56:17] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Operations: ~1 request/minute to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021 (10Ottomata) a:03Ottomata Interesting. So we don't know exactly where the timeout is occurring? Assigning to... [16:04:44] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10elukey) Thanks Ema, really great analysis! I am wondering if we could quickly test how varnishncsa behaves when we pass `-q`, that seems to be the big difference between the tw... [17:36:47] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) >>! In T264074#6536967, @elukey wrote: > I am wondering if we could quickly test how varnishncsa behaves when we pass `-q`, that seems to be the big difference between the... [18:44:39] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10elukey) There is another test that we could do, namely grouping. As far as I can see in the varnishkafka change, the [[ https://github.com/wikimedia/varnishkafka/commit/b0675e80... [22:04:10] 10Traffic, 10Advanced-Search, 10Discovery-Search, 10Operations, and 3 others: Strange URL pattern after search https://en.wikipedia.org/w/index.php?sort=relevance&sort=relevance&sort=relevance&sort=relevance&sort=relevance&sort=relevance ... - https://phabricator.wikimedia.org/T243884 (10Ladsgroup) I think...