[00:41:47] 10Traffic, 6Operations, 10RESTBase, 6Services, 3Mobile-Content-Service: Split slash decoding from general percent normalization in Varnish VCL - https://phabricator.wikimedia.org/T127387#2058357 (10GWicke) > It still seems to me like the only generic decoding we can do for all traffic is the unreserved s... [01:45:46] 7HTTPS, 10Traffic, 6Research-and-Data, 10The-Wikipedia-Library, and 5 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#2058566 (10TheDJ) >>! In T87276#2055795, @BBlack wrote: > Fascinating :) Any idea if it also throws an... [01:51:18] 10Traffic, 6Operations, 10Parsoid, 10VisualEditor, and 2 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#1578558 (10Jackmcbarn) FYI, I didn't realize that this was taking parsoid-prod.wmflabs.org down until it happened, so a lot of the requests from it we... [01:54:28] 10Traffic, 6Operations, 10Parsoid, 10VisualEditor, and 2 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#2058577 (10ssastry) >>! In T110474#2058575, @Jackmcbarn wrote: > FYI, I didn't realize that this was taking parsoid-prod.wmflabs.org down until it hap... [01:55:15] 10Traffic, 6Operations, 10Parsoid, 10VisualEditor, and 2 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#2058578 (10Jackmcbarn) No problem, I already have it taken care of. [04:48:34] 7HTTPS, 10Traffic, 6Operations, 10Wikipedia-Store: shop.wikimedia.org should be HTTPS only - https://phabricator.wikimedia.org/T39790#2058723 (10Dzahn) shop.wikimedia.org and shop.wikipedia.org point to our Apache cluster: ``` templates/wikipedia.org:shop 600 IN DYNA geoip!text-addrs templ... [04:52:14] 7HTTPS, 10Traffic, 6Operations, 10Wikipedia-Store: shop switches HTTPS -> HTTP when showing login prompt (on clicking checkout) - https://phabricator.wikimedia.org/T63528#2058724 (10Dzahn) the shop does not run on WMF infra but external on shopify.com and isn't operated by the operations team. so i'm afrai... [04:54:46] 10Traffic, 10domains, 10DNS, 6Operations: Transfer of domain names to WMF servers - https://phabricator.wikimedia.org/T114922#2058726 (10Dzahn) 5stalled>3Open [04:58:16] 10Traffic, 10domains, 10DNS, 6Operations: Transfer of domain names to WMF servers - https://phabricator.wikimedia.org/T114922#2058727 (10Dzahn) 5Open>3Resolved a:5VBaranetsky>3None We have received no updates on this since November, but it has been fixed at some point. whois wilkipedia.org | grep... [05:00:37] 7HTTPS, 10Traffic, 6Operations, 10Wikipedia-Store: shop switches HTTPS -> HTTP when showing login prompt (on clicking checkout) - https://phabricator.wikimedia.org/T63528#2058731 (10HuiZSF) Seems shopify only use https for checkout transaction. Similar report earlier in last year: https://phabricator.wiki... [08:47:16] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058868 (10Joe) [08:47:25] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058880 (10Joe) p:5Triage>3Unbreak! [08:47:56] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058868 (10Joe) a:3Joe [08:58:24] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058903 (10Joe) So, SAL entries around the time of the two spikes: February 17th: ``` 20:25 logmsgbot: krinkle@tin Synchronized wm... [09:30:23] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058954 (10Joe) I am looking at the number of errors from the text caches on various days, focusing on the affected backend as x_cac... [09:31:32] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058956 (10Joe) [09:37:21] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058981 (10ArielGlenn) An overly long list of puppet stuff before/after the Feb 17th spike [17:20:41] (CR) Dzahn: [C: 2] partman... [09:47:12] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2058996 (10ema) We did enable nginx keepalives for text at 8PM on the 17th https://phabricator.wikimedia.org/T107749#2036351 Could t... [10:00:25] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2059048 (10Joe) The timing of the first spike correlates pretty well with that change; this still doesn't explain the second step up. [10:02:49] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2059053 (10Joe) [10:19:57] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2059068 (10Joe) The second surge is very steep, it happens between 16:03 UTC and 16.:07 UTC on Feb 23rd. This surge speed would see... [11:09:56] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2059157 (10ema) A few interesting varnish metrics in esams [[ https://ganglia.wikimedia.org/latest/?r=custom&cs=02%2F16%2F2016+00%3A... [14:02:04] 10Traffic, 6Operations: Increase request limits for GETs to /api/rest_v1/ - https://phabricator.wikimedia.org/T118365#2059647 (10faidon) a:5faidon>3None [14:18:05] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2059699 (10ema) We have also noticed an increase in varnish.fetch_chunked as well as a varnish.fetch_close drop. {F3412235} {F3412... [15:36:58] 7HTTPS, 10Traffic, 6Operations: SSL cert needed for benefactorevents.wikimedia.org - https://phabricator.wikimedia.org/T115028#2060021 (10RobH) In the past, all fundraising keys/certs are generated/stored and stored on boron, NOT in our ops private/public puppet repos. Since this is being used on a third pa... [15:48:31] 7HTTPS, 10Traffic, 6Operations: SSL cert needed for benefactorevents.wikimedia.org - https://phabricator.wikimedia.org/T115028#2060059 (10Jgreen) Generating/storing in the normal repo sounds good to me. [17:49:49] 7HTTPS, 10Traffic, 10Wikimedia-Fundraising, 7Blocked-on-Fundraising-Tech: links.email.donate.wikimedia.org should offer HTTPS - https://phabricator.wikimedia.org/T74514#2060677 (10Nemo_bis) >>! In T74514#2051351, @CCogdill_WMF wrote: > Sure @Nemo_bis, here's a sample link from an email that went out last w... [18:05:04] 10Traffic, 6Operations, 13Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#2060728 (10BBlack) 5Resolved>3stalled See T127931 - caused elevated (but still small) 503s on cache_text, reverted for now, will revisit after further varnish/nginx so... [18:18:17] 10Traffic, 10MediaWiki-General-or-Unknown, 6Operations, 13Patch-For-Review: Number of 5xx (mainly 503) errors surge from text caches since February 17th - https://phabricator.wikimedia.org/T127931#2060780 (10BBlack) 5Open>3Resolved [18:20:06] 10Traffic, 10MediaWiki-Authentication-and-authorization, 6Operations: Logging out of a wiki leaves an XXwikiSession= Cookie behind - https://phabricator.wikimedia.org/T127436#2060781 (10Anomie) > I'm not 100% sure whether this is a recent regression - I think it is It's not. Logging out in MediaWiki clears... [20:19:16] 7HTTPS, 10Traffic, 6Operations, 10Wiki-Loves-Monuments-General: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2061293 (10Multichill) You can hardly call "Registrar.eu" an individual in Rotterdam or are you looking at different records than I am? * wikilovesmonum... [20:24:33] 10Traffic, 10MediaWiki-Interface, 6Operations, 5MW-1.27-release, and 3 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2061306 (10Sjoerddebruin) Another page: http... [21:34:23] 10Traffic, 10MediaWiki-Authentication-and-authorization, 6Operations: Logging out of a wiki leaves an XXwikiSession= Cookie behind - https://phabricator.wikimedia.org/T127436#2061676 (10Tgr) [[ https://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf | OWASP ASVS ]] 2.7 (a draft... [21:35:35] 10Traffic, 10Continuous-Integration-Infrastructure, 6Operations, 13Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2061679 (10hashar) Verified on Nodepool, requests to Jenkins are all fine: ``... [22:21:41] 10Traffic, 10MediaWiki-Authentication-and-authorization, 6Operations: Logging out of a wiki leaves an XXwikiSession= Cookie behind - https://phabricator.wikimedia.org/T127436#2061751 (10csteipp) Yep, exactly what @Tgr said. Historically, it was considered best practice to delete or change it, in case you for...