[10:59:50] https://dev.to/riknelix/fast-and-efficient-recompression-using-previous-compression-artifacts-47g5 [12:55:30] * andre__ passes two beers to ema [12:55:47] hello :) [12:55:53] heja! [12:56:14] not sure I can really help with this (lacking technical understanding), but worth a try I guess [12:56:48] grep'ing an upstream git checkout of Phab does not give me any matches for X-Forwarded-For or X-Client-IP [12:58:38] ema: I dumped all I know into https://phabricator.wikimedia.org/T237637#6385498 [12:59:16] andre__: so the issue is that phab logs do show the correct IP for IPv4, but not for IPv6? [13:00:26] https://secure.phabricator.com/T10375 "Support X-Forwarded-For in Access Log" [13:00:30] "resolved lolno" [13:02:32] ema: Checking it again, it seems that all and any IP addresses (both IPv4 and IPv6) are the one very same IP now (I think that I'm still on IPv4 but I just logged out and in again and I also get that very common IP) [13:02:39] so seems that it's completely broken by now [13:02:45] ema: Can you access https://phabricator.wikimedia.org/people/logs/ ? [13:03:21] andre__: I can and indeed I see the same IP for every entry [13:03:29] yepp. That's my problem :P [13:03:42] the problem is now clear :) [13:04:03] 8.0.0.0.6.1.0.0.4.6.0.0.0.1.0.0.2.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa name = phab1001.eqiad.wmnet. [13:05:25] X-Client-IP is the request header that phab should use for this [13:06:05] (sounds like some upstream changes needed?) [13:06:17] https://secure.phabricator.com/book/phabricator/article/configuring_preamble/ [13:06:22] >Adjusting Client IPs [13:06:25] >If your install is behind a load balancer, Phabricator may incorrectly detect all requests as originating from the load balancer, rather than from the correct client IPs. [13:06:45] ffff* why haven't I found that one yet, thanks [13:07:15] Might be worth double checking what our preamble is [13:07:41] Presumably if it changed at some point, it coincides with either a phab upgrade or something else infront of phab being updated [13:08:15] Or maybe we're all actually existing in a simulation running on phab1001 [13:10:00] I feel that Reedy is gonna get those beers at the end of the day [13:10:20] I might have to split them into halfs :-/ [13:10:47] but that's already all damn helpful, thanks. Now just need to find that our damn preamble in some repo [13:11:09] I haven't eaten yet, will be back and read the scrollback after lunch! [13:11:57] $preamble = "${phabricator::confdir}/preamble.php" [13:12:04] content => template('phabricator/preamble.php.erb'), [13:12:13] https://github.com/wikimedia/puppet/blob/8841bafe66c888a24f83b226558ba1a3801985e5/modules/phabricator/templates/preamble.php.erb [13:12:56] Also, mukunda just replied on the task [13:13:05] twentyafterfour ^ [13:15:46] andre__: Also, for your grepping.. You'd probably need to s/-/_/ [13:15:59] ah, good point [13:16:20] at least for the PHP code using it rather than the comments you picked up [14:20:00] Reedy, andre__: so, at least on paper, the fix would boild down to setting $_SERVER['REMOTE_ADDR'] = $_SERVER['HTTP_X_CLIENT_IP'] in preamble.php.erb? [14:20:12] s/boild/boil/ [14:20:33] I dunno on the specifics, but it wouldn't surprise me if it's something like that [14:20:37] Probably needs a bit more guarding [14:20:51] mukunda's comment makes it sound like apache used to do this, but seemingly stopped [14:21:59] prior to the apache module we had custom code in PHP via a file installed by puppet called preamble.php [14:22:25] last I remember we didn't need the preamble code anymore [14:23:08] preamble.php is installed by a puppet module (phabricator::redirector) that says: [14:23:11] # Setup the preamble.php and redirect_config.json to redirect bugzilla [14:23:12] /srv/phab/phabricator/support/preamble.php [14:23:14] # (and eventually RT) urls to phabricator [14:23:53] so perhaps this used to do redirection at some point, and then the functionality moved elsewhere? [14:24:17] ema: preamble redirect stuff is still used Afaik [14:24:41] it redirects old bugzilla urls to phab tasks [14:25:03] ack, well if it's still there than it should be very easy to check if setting $_SERVER['REMOTE_ADDR'] there fixes the IP logging issue [14:25:53] preamble handles HTTP_X_FORWARDED_PROTO and it used to handle x-forwarded-for headers but it was removed and replaced with some apache module that translated the remote address automatically [14:26:17] I have to run, not supposed to be working today ;) [14:26:44] twentyafterfour: enjoy :) [16:37:39] and for folks who like traffic issues, see https://phabricator.wikimedia.org/T260440 which is gaining duplicates :) [16:38:21] thanks andre__, known for a while now and a misconfiguration on their side, FWIW, but we should make a tracking task [16:39:14] not sure what the term "tracking task" means if we already have a ticket, but if there's any public info to share or give back to communities it could be nice to add it to that ticket [16:42:05] that is a private security issue with a user IP address in it :) [16:44:07] ah, so you meant public. I see :) [16:50:19] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) [16:50:32] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) [16:51:37] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) For posterity, relevant workaround patch and deployment thereof: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/620377 https://sal.toolforge... [16:52:10] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) [17:00:33] thanks [17:01:10] also, should be fixed for now [17:02:08] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) [17:02:29] 10Traffic, 10netops, 10Operations: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10CDanis) 05Open→03Resolved a:03CDanis There's still an issue on Jio's side that needs to be fixed by them, but, we've put a temporary workaround in place,...