[08:10:03] 10HTTPS, 10Traffic, 10DBA, 10Operations, and 2 others: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499#3419578 (10Aklapper) a:05Springle>03None [ Resetting assignee as assignee account is not active anymore ] [08:48:15] 10Traffic, 10Discovery, 10Maps, 10Operations, and 2 others: What is a reasonable per-IP ratelimit for maps - https://phabricator.wikimedia.org/T169175#3419717 (10ema) So, @mpopov's analysis seems to be based on all requests, including varnish cache hits. On the cache_text cluster, we're currently rate-limi... [10:55:04] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3420148 (10ema) [10:55:14] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3420163 (10ema) p:05Triage>03High [12:19:50] 10Traffic, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420358 (10daniel) @aaron I'm trying to wrap my head around the code in Title::invalidateCache(), bu... [12:21:52] 10netops, 10Operations, 10ops-eqiad: Replace cr1/2-eqiad air filters - https://phabricator.wikimedia.org/T170138#3420362 (10faidon) [12:24:25] 10Traffic, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420378 (10daniel) @jcrespo Looking at T169884, it seems that it's an unrelated issue triggered by... [13:08:55] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3420503 (10Peachey88) [13:15:26] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3420538 (10Jgreen) [13:15:32] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#3420541 (10Jgreen) [13:16:22] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2201391 (10Jgreen) [13:16:41] 10netops, 10Operations, 10monitoring: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144#3420547 (10faidon) [13:16:53] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2201391 (10Jgreen) [13:17:39] 10Traffic, 10Operations: stream.wikimedia.org: remove legacy rcstream/socket.io HTTPS redirect hole punches - https://phabricator.wikimedia.org/T168919#3420564 (10Ottomata) Yup! I just got back from vacation, but baring some unknown blocker, plan to turn it off this week. (still checking emails though!) [13:21:21] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#3420572 (10Jgreen) [13:21:24] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3420570 (10Jgreen) 05Open>03Resolved Closing this task because the remaining non-compliant site, benefactorevents.wikimedia.org, has been shut down. [13:27:37] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3420580 (10ayounsi) Some of those spikes seem to match with OSPF flaps triggered by BFD. The link between cr2-codfw and cr2-eqiad stay up, no packet loss, but for some reasons BFD occas... [13:31:43] 10Traffic, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420592 (10Ladsgroup) a:05Ladsgroup>03daniel [13:59:43] 10Traffic, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420723 (10daniel) @aaron another question: does RefreshLinksJob also purge the CDN cache automatica... [15:42:45] 10Traffic, 10Operations, 10Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#3421257 (10BBlack) Recap of recent progress: where we're at now is a hard cap of 1 day TTL within each cache layer, regardless of any longer max-age sent by the application layer. De... [15:51:06] 10Traffic, 10Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#3421301 (10BBlack) I've updated over in T124954#3421257 on the TTL reduction work that has happened in recent months. The TL;DR there is that even in the case that no purging works at all and the application-spe... [15:57:30] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#3421337 (10BBlack) [15:57:33] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3421335 (10BBlack) 05Resolved>03Open All of these hostnames are still in DNS AFAICS: ``` benefactors.wikimedia.org benefactorevents.wikimedia.org eventdonations.wiki... [16:04:50] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3421371 (10Papaul) Racking and cabling in progress [16:11:25] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3421416 (10Jgreen) [16:11:28] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3421415 (10Jgreen) [16:26:24] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3421463 (10Jgreen) >>! In T137161#3421335, @BBlack wrote: > All of these hostnames are still in DNS AFAICS: > > ``` > benefactors.wikimedia.org > benefactorevents.wikime... [16:28:00] 10Traffic, 10netops, 10Operations, 10ops-eqiad: Upgrade BIOS/RBSU/etc on lvs1007 - https://phabricator.wikimedia.org/T167299#3421466 (10BBlack) Update? [17:08:58] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3421623 (10ayounsi) About naming, the "issue" is that the devices are stacked, so logically seen as a single device. It happen when two stacked/clustered devices are... [17:36:30] 10Traffic, 10Operations, 10Community-Liaisons (Jul-Sep 2017): Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3421794 (10BBlack) Sorry, we've been wanting to make forward progress on this for several months... [18:27:11] 10Traffic, 10Discovery, 10Maps, 10Operations, and 2 others: What is a reasonable per-IP ratelimit for maps - https://phabricator.wikimedia.org/T169175#3422063 (10BBlack) +1 on using a similar rate to the APIs on text. I wonder what the peak (ab?)users' rates on `upload.wikimedia.org` look like as well, an... [18:28:37] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3422073 (10BBlack) benefactors - It wasn't originally part of the original task here, we've just been questioning whether it's also being removed at the same time, since... [18:35:43] 10Traffic, 10Discovery, 10Maps, 10Operations, and 2 others: What is a reasonable per-IP ratelimit for maps - https://phabricator.wikimedia.org/T169175#3422107 (10Pnorman) > I'm not sure I agree with this conclusion? A Pokemon Go fansite using our tiles making 349 req/sec would not be rate-limited? If we se... [20:30:59] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3422630 (10Peachey88) [20:31:10] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: remove eventdonations.wikimedia.org CNAME - https://phabricator.wikimedia.org/T170192#3422631 (10Peachey88) [23:15:51] 10Traffic, 10AbuseFilter, 10Operations, 10Zero: user_wpzero doesn't always work - https://phabricator.wikimedia.org/T169907#3423257 (10Legoktm) The code is as simple as: ```lang=php $vars->setVar( 'user_wpzero', $wgRequest->getHeader( 'X-Carrier' ) !== false ); ``` So maybe that header isn't alw... [23:43:22] 10Traffic, 10AbuseFilter, 10Operations, 10Zero: user_wpzero doesn't always work - https://phabricator.wikimedia.org/T169907#3423355 (10Legoktm) 05Open>03Invalid I looked up the IPs the users used in the database and according to @bawolff, they're not Wikipedia Zero IP's.