[08:20:48] https://news.ycombinator.com/item?id=14367205 [08:40:24] South Africa is probably the most obvious choice, Cloudflare e.g. has it's African CDN nodes in Cape Town and Johannesburg as well [10:32:34] 10Traffic, 10Analytics, 06Operations, 15User-Elukey: Update Varnishkafka to support TLS encryption/authentication - https://phabricator.wikimedia.org/T165736#3275487 (10elukey) [10:33:55] 10Traffic, 10Analytics, 06Operations, 15User-Elukey: Update Varnishkafka to support TLS encryption/authentication - https://phabricator.wikimedia.org/T165736#3275506 (10elukey) [10:46:44] 10Traffic, 10Analytics, 10Analytics-Cluster, 06Operations, 15User-Elukey: Encrypt Kafka traffic, and restrict access via ACLs - https://phabricator.wikimedia.org/T121561#3275543 (10elukey) [14:04:08] 10Traffic, 06Operations, 10Pybal: Fully-redundant LVS clusters using Pybal per-service MED feature - https://phabricator.wikimedia.org/T165764#3276237 (10BBlack) [14:20:52] 10Traffic, 06Operations: Refactor pybal/LVS config for shared failover - https://phabricator.wikimedia.org/T165765#3276305 (10BBlack) [14:21:39] 10Traffic, 06Operations: Refactor pybal/LVS config for shared failover - https://phabricator.wikimedia.org/T165765#3276336 (10BBlack) [14:35:26] paravoid: in the above, I think I'm probably thinking too abstractly at present, because there's also a router-redundancy issue? (as in, each pybal/lvs only connects to 1/2 routers) [14:35:56] (in interview) [14:36:55] eh ignore that anyways, it's not really an issue I don't think [14:37:52] we can just put the primary hosts on cr1 and the "spare" LVS on cr2 [14:38:07] like we do now, just fewer spares [14:38:34] although that kind of implies that if we lose cr1, we're also crunching down to 1x LVS (which I assume actually works in all cases, but it hasn't actually been tested) [14:39:05] it would be niftier if pybal could just directly advertise to both [14:42:35] it doesn't look like there's any fundamental reason it couldn't advertise to both, actually [14:43:01] just the main pybal.py code happens to read config for a single peer and create objects and such for a single peer, but it seems like it could do more [14:50:46] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276493 (10jcrespo) 05declined>03Open This just happened again on s4. [14:52:39] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3224448 (10Marostegui) Some graphs that were shown while troublshooting https://grafa... [14:54:30] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276528 (10jcrespo) p:05Low>03Triage This is probably not user-requested invalidat... [15:16:30] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276584 (10jcrespo) Without entering on heavy rearchitectures, we should, an probably... [15:26:40] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 06Operations, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#3276610 (10BBlack) 05Resolved>03Open Not resolved, as the purge graphs can attest! [15:26:43] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#3276612 (10BBlack) [15:37:17] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276650 (10jcrespo) [16:01:58] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276828 (10jcrespo) Lots of category pages invalidations happening at that time: ``` UPDATE /* Title::invalidateCache */ `page` SE... [16:04:54] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276894 (10jcrespo) For the long term: how useful is this field, and could it be separated from the rest of the table if it happens t... [19:10:18] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in HTTPS/HSTS configs in wikimedia.org domain - https://phabricator.wikimedia.org/T137161#3277249 (10Jgreen) [19:10:44] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in HTTPS/HSTS configs in wikimedia.org domain - https://phabricator.wikimedia.org/T137161#2359459 (10Jgreen) [19:11:22] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#2359459 (10Jgreen) [19:13:12] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3277260 (10Jgreen) a:05Jgreen>03None This isn't something fr-tech-ops can fix, it's an external site. [19:19:43] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3277275 (10Krinkle) [19:56:13] 10Traffic, 10DBA, 06Operations, 06Performance-Team: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3277390 (10aaron) The query does not come from HTMLCacheUpdateJob (which calls HTMLCacheUpdateJob::invalidateTitles) or seemingly any... [20:02:49] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 06Operations, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#3277398 (10aaron) a:05aaron>03None [20:03:57] 10Traffic, 10DBA, 06Operations, 06Performance-Team, 10Wikidata: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3277400 (10aaron)