[00:58:54] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#3423500 (10BBlack) [00:58:57] 10Traffic, 10Operations: stream.wikimedia.org: remove legacy rcstream/socket.io HTTPS redirect hole punches - https://phabricator.wikimedia.org/T168919#3423497 (10BBlack) 05Open>03Resolved a:03BBlack This hole was removed today in https://gerrit.wikimedia.org/r/#/c/364252 , so this is resolved assuming w... [01:00:53] 10Traffic, 10Operations: stream.wikimedia.org: remove legacy rcstream/socket.io HTTPS redirect hole punches - https://phabricator.wikimedia.org/T168919#3423524 (10BBlack) [04:09:36] 10Traffic, 10Operations, 10Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#3423643 (10Krinkle) >>! In T124954#3421257, @BBlack wrote: > [..] We don't believe it should be possible at this time for an object to exist in the caching layers for more than 4 days... [04:09:45] 10Traffic, 10Operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#3423644 (10Krinkle) [08:41:33] 10Traffic, 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3424125 (10aaron) >>! In T164173#3420723, @daniel wrote: > @aaron another question: does RefreshLink... [13:45:28] ema: while composing a response to Max above, I had the semi-related realization that we might not be handling "keep" correctly in the upload case. Because we're doing that whole hack of "try to limit bad-304 fallout by capping keep to ttl", but swift doesn't really send us TTLs and we default-set them to 1d in the upload case, so keep ends up stuck at 1d too. [13:45:32] I think [15:06:27] 10Traffic, 10Operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#3425280 (10BBlack) >>! In T124954#3423643, @Krinkle wrote: >>>! In T124954#3421257, @BBlack wrote: >> [..] We don't believe it should be possible at this time for an object to exist in the caching layers f... [16:08:02] 10netops, 10Cloud-Services, 10Cloud-VPS, 10Operations, 10ops-codfw: codfw: labtestpuppetmaster2001 switch port configuration - https://phabricator.wikimedia.org/T167321#3425572 (10RobH) a:05RobH>03Papaul Assigned to Papaul to try another NIC on the server, and open a support case for a bad nic if so. [16:12:54] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425597 (10RobH) If the old certificate was not compromised, it is a lot cleaner to simply let it expire. Revokcation, as I understanding... [16:13:56] 10netops, 10Cloud-Services, 10Cloud-VPS, 10Operations, 10ops-codfw: codfw: labtestpuppetmaster2001 switch port configuration - https://phabricator.wikimedia.org/T167321#3425612 (10Papaul) @Robh this is already done it was not switch problem it was DNS see T167157 [16:14:27] 10netops, 10Cloud-Services, 10Cloud-VPS, 10Operations, 10ops-codfw: codfw: labtestpuppetmaster2001 switch port configuration - https://phabricator.wikimedia.org/T167321#3425614 (10RobH) 05Open>03Resolved [16:15:27] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425617 (10BBlack) I think in this case we should revoke unless the expiry is already very close (it might be!). This is private key that... [16:16:45] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425630 (10BBlack) Ah I missed the part above where it stated that it expired in a week or two. In that case, there's little point for this... [16:17:35] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3422468 (10faidon) Looks like it expires in September: ``` Validity Not Before: Jul 18 18:16:03 2016 GMT No... [16:19:18] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425657 (10RobH) I only advised against revokcation since that was my understanding from @bblack, I'm not trying to block this. In fact, I... [16:21:59] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425669 (10RobH) a:03RobH Chatted in irc, I'll revoke this shortly. [16:23:54] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3425704 (10Jgreen) [16:24:22] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3425709 (10RobH) a:03RobH [16:26:38] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#3425715 (10BBlack) [16:28:18] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: remove eventdonations.wikimedia.org CNAME - https://phabricator.wikimedia.org/T170192#3425724 (10RobH) [16:28:21] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425722 (10RobH) 05Open>03stalled Certificate Status: Revoke Processing on Globalsign's systems. I'm going to move this to stalled, a... [16:28:40] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425727 (10RobH) p:05Triage>03Normal [16:33:28] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: remove eventdonations.wikimedia.org CNAME - https://phabricator.wikimedia.org/T170192#3425768 (10RobH) 05Open>03Resolved a:03RobH dns removed [16:47:06] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: remove eventdonations.wikimedia.org CNAME - https://phabricator.wikimedia.org/T170192#3425918 (10RobH) [16:47:09] 10Domains, 10Traffic, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: revoke eventdonations.wikimedia.org SSL cert if there is one... - https://phabricator.wikimedia.org/T170193#3425916 (10RobH) 05stalled>03Resolved Revocation Request Completed for eventdonations.wikimedia.org [16:55:33] 10Traffic, 10Operations: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3425957 (10RobH) Please note benefactorevents.wikimedia.org doesn't expire until 04/02/2018. Since this private key is accessible by a third party (Trilogy), I'm revoking the certificate... [17:03:24] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3425988 (10RobH) [17:03:27] 10Traffic, 10Operations, 10Patch-For-Review: revoke benefactorevents.wikimedia.org SSL certificate - https://phabricator.wikimedia.org/T170140#3425984 (10RobH) 05Open>03Resolved Revocation Request Completed for benefactorevents.wikimedia.org - confirmed from globalsign. I've gone ahead and removed the k... [17:04:24] 10Traffic, 10DNS, 10Operations, 10fundraising-tech-ops: remove benefactorevents.wikimedia.org cname from DNS - https://phabricator.wikimedia.org/T170295#3425995 (10Jgreen) [17:05:27] 10Traffic, 10DNS, 10Operations, 10fundraising-tech-ops: remove benefactorevents.wikimedia.org cname from DNS - https://phabricator.wikimedia.org/T170295#3426010 (10Jgreen) 05Open>03Resolved p:05Triage>03Normal a:03RobH [17:10:51] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#3426051 (10BBlack) [17:10:54] 10Traffic, 10Operations: Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains - https://phabricator.wikimedia.org/T137161#3426050 (10BBlack) 05Open>03Resolved [17:11:52] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2201391 (10BBlack) [17:11:56] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#3426063 (10BBlack) [17:11:59] 10HTTPS, 10Traffic, 10Operations, 10Wikimedia-Shop: store.wikimedia.org HTTPS issues - https://phabricator.wikimedia.org/T128559#3426061 (10BBlack) [17:12:27] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#1423897 (10BBlack) [17:12:31] 10HTTPS, 10Traffic, 10Operations: Enable HSTS on Wikimedia sites - https://phabricator.wikimedia.org/T40516#3426075 (10BBlack) [17:12:34] 10HTTPS, 10Traffic, 10Operations, 10Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2201391 (10BBlack) 05Open>03Resolved a:03BBlack Resolving this and moving the last remaini... [17:12:43] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3426077 (10Papaul) Racking complete [17:14:32] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#3426099 (10BBlack) So with these changes and cleanups in the past few weeks, we're basically down to two outstanding issues here from the original context: * T133548 - Create... [17:16:04] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#3426110 (10BBlack) [17:17:14] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#1423900 (10BBlack) [17:19:08] 10HTTPS, 10Traffic, 10Operations, 10Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#3426128 (10BBlack) [17:29:36] 10HTTPS, 10Traffic, 10Operations, 10Wikimedia-Shop: store.wikimedia.org HTTPS issues - https://phabricator.wikimedia.org/T128559#3426177 (10BBlack) [20:53:08] 10Traffic, 10Varnish, 10Cloud-Services, 10Operations, and 3 others: Fix RESTBase support for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T102178#3427614 (10GWicke) [20:53:48] 10Traffic, 10Operations, 10Services (watching), 10discovery-system, 10services-tooling: Figure out an etcd deploy strategy that includes multi DC failure scenarios. - https://phabricator.wikimedia.org/T98165#3427621 (10GWicke) [22:30:00] 10Traffic, 10Operations, 10RESTBase, 10RESTBase-API, 10Services (next): RESTBase support for www.wikimedia.org missing - https://phabricator.wikimedia.org/T133178#3428189 (10GWicke) [22:53:59] 10Traffic, 10MediaWiki-API, 10Operations, 10monitoring, 10Services (watching): Set up action API latency / error rate metrics & alerts - https://phabricator.wikimedia.org/T123854#3428308 (10GWicke) [23:05:18] 10Traffic, 10Citoid, 10ContentTranslation, 10ContentTranslation-CXserver, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#3428358 (10GWicke) So at this point basically only cxserver is remaining. Work on that is ongoing... [23:05:46] 10Traffic, 10Citoid, 10ContentTranslation, 10ContentTranslation-CXserver, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#3428360 (10GWicke)