[17:13:55] <wikibugs>	 3Wikimedia-General-or-Unknown, MediaWiki-Core-Team: Existed pages without ability to reach and obviously wrong namespace - https://phabricator.wikimedia.org/T87645#1023624 (10Glaisher)
[19:59:15] <_joe_>	 ori: see the SAL, I think we updated it on tuesday
[19:59:35] <_joe_>	 it was just the XOR patch and the PCRE cache
[19:59:40] <_joe_>	 well, "just"
[20:39:42] <ori>	 _joe_: https://phabricator.wikimedia.org/T88942
[20:41:00] <_joe_>	 ori: well, there were two tickets, one opened by you, requiring those patches to be deployed
[20:41:21] <_joe_>	 (which blocked quite a few other bugs too)
[20:41:25] <ori>	 yeah, i'm not blaming
[20:41:31] <ori>	 it may not even be related, actually
[20:41:33] <ori>	 this is just a hunch
[20:41:38] <_joe_>	 it probably is
[20:42:48] <_joe_>	 btw, I didn't notice it after deploying 2 weeks ago to beta, and 1 week later on the canary pool. Maybe exceptions don't get recorded in /var/log/hhvm/error.log?
[20:43:09] <ori>	 possibly. i have to admit i don't always remember what type of error goes where. these are exceptions, not fatals.
[20:43:17] <ori>	 i.e., hhvm keeps running.
[20:43:24] <ori>	 bbiaf
[20:43:52] <ori>	 it's your weekend but tim's monday starts in a couple of hours so we can ask him to look
[20:44:35] <_joe_>	 sure
[20:45:04] <_joe_>	 interestingly enough, https://en.wikipedia.org/w/index.php?title=College_of_Arms&action=edit works
[20:45:07] <_joe_>	 !!!!
[20:45:08] <_joe_>	 wtf
[20:45:33] <_joe_>	 this does not: https://en.wikipedia.org/w/wiki.phtml?title=College_of_Arms&action=edit
[20:50:24] <ori>	 phtml?
[20:51:03] <ori>	 https://en.wikipedia.org/wiki/College_of_Arms?action=edit works as well
[20:51:14] <ori>	 /w/wiki.phtml is funny, how did you devise that URL?
[20:51:31] <ori>	 nevertheless, it's very interesting. is it getting handled by plain PHP somehow?
[20:53:08] <ori>	 _joe_: https://en.wikipedia.org/wiki/College_of_Arms?action=edit (which works) has header: X-Powered-By: HHVM/3.3.1
[20:53:24] <ori>	 https://en.wikipedia.org/w/wiki.phtml?title=College_of_Arms&action=edit (which doesn't) has header: X-Powered-By: HHVM/3.3.0-static
[20:53:37] <_joe_>	 oh my
[20:53:55] <_joe_>	 so it's interpreted by php.
[20:54:14] <_joe_>	 that's why we did not catch it on the canary pool where we removed mod_php
[20:54:17] <ori>	 do you think? by then why 3.3.0 vs 3.3.1?
[20:55:06] <_joe_>	 that's the header we added to all content non-from-HHVM to use in the varnish cache layer separation
[20:55:10] <_joe_>	 that should go btw
[20:55:15] <ori>	 oh, i see
[20:55:17] <_joe_>	 I'm pretty sure
[20:55:30] <_joe_>	 I already corrected one bug like this
[20:55:34] * _joe_ headdesks
[20:55:51] <ori>	 right: modules/mediawiki/files/apache/configs/hhvm_mark_engine.conf:3:    Header always setifempty X-Powered-By "HHVM/3.3.0-static"
[20:55:57] <_joe_>	 so I'm pretty sure that url will give a 404 on the canary hosts
[20:56:12] <_joe_>	 try that request on mw1019
[20:56:19] <_joe_>	 and on mw1030
[20:56:24] <_joe_>	 the first will be a 404
[20:56:28] <_joe_>	 (no mod_php)
[20:56:32] <ori>	 this explains why an error that looks so severe is happening only once every 10 mins or so and is not accompanied by loud user complaints
[20:56:36] <ori>	 because it's happening on 404s
[20:56:41] <_joe_>	 yes
[20:56:45] <ori>	 that's good news, it means it hasn't impacted editors in a serious way
[20:56:55] <_joe_>	 well, it's still caught by the old php
[20:57:06] <_joe_>	 so maybe it was a "valid" url before
[20:57:21] <_joe_>	 mh, it most surely wasn't
[20:57:30] <ori>	 yeah, i don't think it was
[20:57:52] <_joe_>	 I have to think of this when I'm not so tired and sad from patching php code all day 
[20:58:12] <ori>	 well, i think it's perfectly okay to leave it, now that we know the impact is very small
[20:58:19] <ori>	 i can update the ticket with what we discovered
[20:58:30] <_joe_>	 next time someone complains about the mediawiki code quality, make them write wordpress for one week
[20:58:41] <ori>	 oh, right! how was the weekend php hacking?
[20:58:41] <_joe_>	 ok thanks
[20:58:50] <_joe_>	 horrible :P
[20:58:51] <ori>	 this is for austici(sp?)
[20:58:54] <_joe_>	 yes
[20:58:58] <ori>	 cool cool
[21:00:08] <_joe_>	 it's a relief this wasn't so serious
[21:48:15] <wikibugs>	 3operations, MediaWiki-Core-Team: Store unsampled API and XFF logs - https://phabricator.wikimedia.org/T88393#1023892 (10Andrew) p:5Triage>3Normal
[22:19:00] <Nemo_bis>	 wiki.phtml are really ancient URLs; it still happens to find some around and click them
[22:19:13] <Nemo_bis>	 BUt it's very rare