[01:21:44] 10Traffic, 10Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2875043 (10matmarex) [01:22:02] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875030 (10matmarex) [01:34:36] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875030 (10Betacommand) I would say this is functioning as intended all /wiki urls are in the short url form. Short urls do not handle any URL params. index.php is the corre... [01:43:39] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875091 (10Writ_Keeper) Fair enough, I suppose--I was surprised that redirect=no worked with the short form at all--but then, does that make the fact that it *does* work a b... [02:03:03] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875030 (10PrimeHunter) >>! In T153275#2875073, @Betacommand wrote: > Short urls do not handle any URL params. It nearly always works fine, e.g.: https://en.wikipedia.org/w... [02:03:35] 10netops, 06Operations, 10ops-eqiad: asw-a2-eqiad PEM 0 not powered - https://phabricator.wikimedia.org/T153273#2874808 (10Cmjohnson) A fuse blew on the PDU...xy phase 2 is down. I am stopping at the store tomorrow to get a SC20 fuse to replace it. [02:26:13] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875030 (10Unready) >>! In T153275#2875117, @PrimeHunter wrote: > It nearly always works fine, e.g.: > https://en.wikipedia.org/wiki/Example?action=history > https://en.wiki... [04:37:06] 10Traffic, 10Varnish, 06Discovery, 06Operations, 06Maps (Tilerator): Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2875353 (10Yurik) [04:40:13] 10Traffic, 06Discovery, 06Operations, 06Maps (Kartographer): Clarify caching to enable direct Wikidata Query Service access by - https://phabricator.wikimedia.org/T146832#2875409 (10Yurik) [07:09:32] as an experiment, I'm running the new varnishxpcs (https://gerrit.wikimedia.org/r/326965) on cp3033 [07:10:24] systemd is neatly showing both processes in the same cgroup: [07:10:25] CGroup: /system.slice/varnishxcps.service [07:10:25] ├─140839 python /usr/local/bin/varnishxcps --statsd-server=statsd.eqiad.wmnet:8125 [07:10:28] └─140841 /usr/bin/varnishncsa -n frontend -q ReqMethod ne "PURGE" -F %{X-Connection-Properties}i [07:12:03] I've compared the stats produced by the new varnishxcps with the old one (with output to stdout instead of statsd) and they match [07:12:58] so now I'll let it run for a while on cp3033, let's see if the statsd stats look good and nothing breaks [07:50:48] looks good! merged and created this to see if there's anything odd https://grafana.wikimedia.org/dashboard/db/http-2?from=now-6h&to=now [08:04:37] cpu usage is about 2.8% varnishxcps and ~ 4.2% varnishncsa which is a huge win compared to the python-varnishapi version [08:05:10] varnishreqstats is happily maxing out one core at the moment for instance [08:09:32] https://grafana.wikimedia.org/dashboard/db/tls-ciphers also doesn't seem to be broken, happy times [08:30:18] 10Wikimedia-Apache-configuration, 06Operations: URL to pagenames with special characters fail - https://phabricator.wikimedia.org/T153275#2875661 (10Writ_Keeper) @Unready: this bug actually applies to all URL parameters, not just redirect, given a page name with a question mark in it. For example: https://en.... [08:31:03] nice! [09:29:02] <_joe_> ema: would you mind taking a look at https://gerrit.wikimedia.org/r/327465 ? [09:29:15] <_joe_> take your time, I'm out for the next ~ 1.5 hours [10:29:47] _joe_: LGTM [11:21:40] elukey: ema: deployment-copper.deployment-prep.eqiad.wmflabs is out of disk space somehow :( [11:21:49] not sure what that instance is used for / represents [11:22:02] WARNING: deployment-prep.deployment-copper.diskspace.root.byte_percentfree (<44.44%) [11:22:26] argh [11:22:44] what is it for ? [11:23:06] building debian packages with more ram than copper (the prod host) [11:23:06] on labs, usually we mount the instance extended disk space under /srv and vary whatever needs to be to point to /srv instead of /var/xx/yyy [11:23:10] oh [11:23:26] CI has some logic to build packages as well :D [11:23:33] yeah it is a try since I have a build that consumes more than 8GB of ram :( [11:23:39] oh man [11:23:53] ema is helping me to figure out where the build breaks [11:24:33] ah I see role::package::builder [11:24:46] I think the CI instances use the same role [11:24:50] but get role::labs::lvm::srv applied [11:25:00] and some tweak to move /var/cache/pbuilder to /srv/pbuilder [11:25:02] or something like [11:25:08] that and hence benefit from more disk space [11:25:31] thanks for the tip :) [11:25:44] though the CI directly call ::package_builder [11:26:19] elukey: modules/contint/manifests/package_builder.pp has all the lame logic [11:26:37] namely get the extended disk mounted before [11:26:48] on /mnt (but really now it is /srv we got a cherry pick) [11:26:55] BTW the build breaks everywhere I'm not sure I've ever seen such a bad package on this planet [11:27:25] then class { '::::package_builder': basepath => '/srv/pbuilder', require => [ Mount['/srv'], File['/srv/pbuilder'] } [11:27:43] and you get moar space [11:30:08] https://clickhouse.yandex ? :D [11:31:09] yeah, we are trying to make it build.. [11:31:31] I have to say that upstream is super happy to improve the package [11:31:35] get their SAAS offer instead? :D [11:31:43] ahhahah [11:32:18] no we need to use clickhouse as alternative to druid for the Analytics Edit History reconstruction project [11:32:57] you know what. One day we will have MagicDB that will be the unified unicorn of everything [11:33:59] UnicornDB [11:34:14] I created the repo in Gerrit [11:34:17] ;D [11:34:38] elukey: I've commented out half of debian/rules and now it builds :P [11:35:29] elukey: the executable is in clickhouse-server-base [11:36:26] 6 files changed, 27 insertions(+), 36 deletions(-) [11:36:35] ahahahh [11:37:37] good luck :} [17:47:49] 10Traffic, 06Operations: String query string in varnish upload - https://phabricator.wikimedia.org/T153336#2877339 (10fgiunchedi) [17:48:04] 10Traffic, 06Operations: Strip query string in varnish upload - https://phabricator.wikimedia.org/T153336#2877354 (10fgiunchedi) [17:56:38] godog: https://gerrit.wikimedia.org/r/#/c/327553/ ? [18:04:15] bblack: yep that seems like it'd do the trick [19:06:31] 10netops, 06Operations, 10ops-eqiad: asw-a2-eqiad PEM 0 not powered - https://phabricator.wikimedia.org/T153273#2877751 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson Replaced both fuses on the B side on that phase. Both sides are powered. The alarm cleared cmjohnson@asw-a-eqiad> show chassis alarms... [20:17:33] 10Traffic, 06Operations, 07HHVM, 13Patch-For-Review, and 2 others: Enable TLS termination on the MediaWiki clusters - https://phabricator.wikimedia.org/T153042#2878050 (10Joe) 05Open>03Resolved [22:19:46] 10Traffic, 06Operations, 10media-storage: Unexplained increase in thumbnail 500s - https://phabricator.wikimedia.org/T147648#2878517 (10Gilles) [22:19:51] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Performance-Team, and 4 others: Mediawiki thumbnail requests for 0px should result in http 400 not 500 - https://phabricator.wikimedia.org/T147784#2878516 (10Gilles) 05Open>03Resolved