[12:08:09] <_joe_> anyone knows why loading facts is so slow in remote datacenters, e.g. drmrs? [12:08:35] <_joe_> ah no it's actually the catalog compilation, which makes more sense as it needs to submit the facts [12:08:44] <_joe_> which I expect to be slow [16:37:37] denisse: this patch is currently undeployed https://gerrit.wikimedia.org/r/c/performance/excimer-ui-server/+/1179124/1 - it seems that given the host runs php7.4, this may be a good thing? [16:59:31] I guess php8.3 is coming with T353912 [16:59:32] T353912: Observability Bookworm upgrades - https://phabricator.wikimedia.org/T353912 [17:18:18] papaul: are your changes from https://gerrit.wikimedia.org/r/1192171 good to get puppet-merged? [17:21:43] jhathaway: cwhite: would either of you be comfortable providing a second pair of eyes reviewing https://gerrit.wikimedia.org/r/1192171? [17:22:42] it has been merged, but not puppet-merge'd and I'm not quite sure I'd like to unilaterally pull it into production without a second pair of eye [17:22:49] *eyes [17:25:56] swfrench-wmf: yes [17:26:14] papaul: ah, great - thanks! [17:26:27] swfrench-wmf: thanks for checking [17:27:29] j.hathaway: c.white: never mind, we're good :) [18:10:17] fabfur: I noticed that as last week, there are requests in webrequest and pageview_actor in Hadoop with uppercase hostnames like MEDIAWIKI.ORG or JA.M.WIKIPEDIA.org. Last week, I wasn't surprised. Today, seeing https://gerrit.wikimedia.org/r/c/operations/puppet/+/1191010 and T392880, I am. Thoughts? [18:10:18] T392880: Move host normalization to haproxy - https://phabricator.wikimedia.org/T392880 [18:10:31] I noticed as recently as last week* - not starting last week [18:12:08] that is surprising https://docs.haproxy.org/2.8/configuration.html#7.3.1-host_only [18:12:12] >This converter also sets the string in lowercase. [18:12:49] https://w.wiki/FUiY [18:14:00] I found it when I was writing the unified mobile domain routing progress report. My query for mobile page views sans ~ .m. had non-zero results even before we rolled out, whereas page views on the standard domain should have been impossible prior to Sept 3. [18:14:03] Krinkle: https://w.wiki/FUia [18:14:22] It was very small numbers but not zero. turned out they eluded the .m. LIKE query because they were .M. [18:14:49] Worked around it with ` NOT RLIKE '(?i)` at https://phabricator.wikimedia.org/T405429#11221830 [18:14:54] but was def a head scratcher for a bit [18:15:32] sorry, I had the order inverted because I started with all requests and looked for rare ones. Thanks [18:49:39] Krinkle: it's strange, as cdanis said haproxy should take care of it, and the patch has been merged only this morning [18:50:44] yeah, the logs in hadoop are noadays based on haproxy rather than varnishkafka so that shouldn;t have changed anything and afaik it didn't, this is an issue I stumbled upon last week and goes back several months [18:50:50] e.g. incl for Sep 1 data [18:51:09] however it is possible that this was working in Varnish/varnishkafka and regressed in the move to haproxy.