[00:06:55] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2970912 (10GWicke) @Tgr: That is great news. This means that we already have a means of feeding generic key-value parameters to the thumb machinery. Are t... [01:16:10] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2971258 (10Tgr) >>! In T66214#2970912, @GWicke wrote: > Are those key-value parameters already supported by all media handlers, or is support limited to b... [01:21:45] 10Traffic, 06Operations, 13Patch-For-Review: convert archiva to use Letsencrypt for SSL cert (deadline 2017-05-08) - https://phabricator.wikimedia.org/T154942#2971281 (10RobH) a:03RobH [01:29:39] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2971305 (10RobH) [01:42:34] 10netops, 06Operations, 10ops-codfw: codfw: mc2019-mc2036/switch port configuration - https://phabricator.wikimedia.org/T156212#2971314 (10RobH) 05Open>03Resolved all have had the port description set, enabled, and proper vlan (internal) set. [07:48:57] 10netops, 10DBA, 06Operations, 13Patch-For-Review: Switchover s1 master db1057 -> db1052 - https://phabricator.wikimedia.org/T156008#2971613 (10Marostegui) This has happened already. Times in UTC: Preparation of all the code, topology changes etc: 06:30-07:30 read only on: 07:30:40 do all the necessary c... [09:11:59] 10netops, 10DBA, 06Operations, 13Patch-For-Review: Switchover s1 master db1057 -> db1052 - https://phabricator.wikimedia.org/T156008#2971729 (10Marostegui) recap of the cleanup work: dns changed for s1-master.eqiad.wmnet multisource slaves changed (only pending dbstore1001): db1047, db1069,dbstore1002 rep... [10:01:09] 10netops, 10DBA, 06Operations, 13Patch-For-Review: Switchover s1 master db1057 -> db1052 - https://phabricator.wikimedia.org/T156008#2971915 (10jcrespo) only pending: * change dbstore1001 to replicate from db1052 [10:59:13] 10netops, 06Labs, 06Operations: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2972065 (10Marostegui) Hi, The pending work of: T156008 shouldn't be a blocker to replace the switch. The switchover was done, and only pending to move dbstore1001 to replicate f... [14:39:42] 10Traffic, 06Operations, 06Operations-Software-Development, 10Pybal: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#2972682 (10ema) >>! In T134893#2950312, @Volans wrote: [...] > Jan 12 13:32:19 lvs2003 pybal[23011]: [pybal] ERROR:... [15:01:32] 10Traffic, 06Operations: Select site vendor for Asia Cache Datacenter - https://phabricator.wikimedia.org/T156030#2972767 (10BBlack) [15:03:45] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2972776 (10BBlack) [15:03:48] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: make blog links from wmfwiki front page use HTTPS links - https://phabricator.wikimedia.org/T104728#2972777 (10BBlack) [15:03:51] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2972773 (10BBlack) 05Open>03Resolved a:03BBlack Confirmed correct current operation: 1) All HTTP access seems to redirect to HTTPS 2) All HTTPS requests send response header: `str... [15:03:58] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2972778 (10BBlack) [15:09:55] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: store.wikimedia.org HTTPS issues - https://phabricator.wikimedia.org/T128559#2972787 (10BBlack) Any updates here? What we're asking for here is a modern HTTPS-only configuration. I'd think an e-commerce vendor would be all about that... [15:10:55] 10Traffic, 06Operations, 10fundraising-tech-ops: Fix nits in Fundraising HTTPS/HSTS configs in wikimedia.org domain - https://phabricator.wikimedia.org/T137161#2972789 (10BBlack) What about benefactorevents / eventdonations? [15:40:33] bblack, ema: I'm currently building openssl 1.1.0d packages, the security issues fixed are minor, but it brings a ton of general bugfixes as well. when I'm done I'll install them on cp1008 for some initial testing [15:41:34] moritzm: thanks! I read the advisory, I'll probably regen our DHE params after it's installed. [15:42:00] moritzm: nice, if testing goes well we can then upgrade together with the kernel and jessie point release [15:42:43] well, now that I say that: our CLI stuff is 1.0.2 still, so regen DHE after that is upgraded too [15:47:59] 1.0.2 will also be upgraded to 1.0.2k tomorrow, the 1.1 build turned out to be somewhat obnoxious with some newly introduced versioned symbols are dh_gensymbolds bailing on that, but I'm somewhat optmistic that my current 1.1 build will work fine [15:48:34] yeah that whole thing with how debian wants the symbol list and upstream doesn't provide it is annoying [15:49:04] number of backport 1.0.2 bugfixes is also pretty big (40 or so), really nice that they care about backporting to older maintenance branches [15:55:20] *grr*, failed again, turns out both libssl _and_ libcrypto have new symbols... [15:59:21] 10Traffic, 06Operations: convert stream.wikimedia.org from GS to LE certificate - https://phabricator.wikimedia.org/T155524#2972927 (10BBlack) stream.wikimedia.org is part of cache_misc now, so if we have an expiring certificate here, I don't think we need to replace it. [16:03:12] 10Traffic, 06Analytics-Kanban, 10EventBus, 06Operations, and 2 others: Productionize and deploy Public EventStreams - https://phabricator.wikimedia.org/T143925#2972976 (10BBlack) cache_misc for this are all implemented and live now. The [[ https://github.com/wikimedia/operations-puppet/blob/production/mod... [16:07:43] 10Traffic, 06Analytics-Kanban, 10EventBus, 06Operations, and 2 others: Productionize and deploy Public EventStreams - https://phabricator.wikimedia.org/T143925#2972982 (10Ottomata) YESSSSSSSSSSSSSSSSS awesome! Thank you! [16:14:30] 10netops, 06Operations: pfws not on librenms - https://phabricator.wikimedia.org/T156381#2973000 (10ema) [16:29:00] 10Traffic, 06Operations: convert stream.wikimedia.org from GS to LE certificate - https://phabricator.wikimedia.org/T155524#2973059 (10Dzahn) https://gerrit.wikimedia.org/r/#/c/334207/ [16:43:21] 10netops, 10Analytics, 06Operations, 13Patch-For-Review: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2973097 (10Nuria) [16:49:59] 10netops, 10Analytics, 06Operations, 13Patch-For-Review: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2973154 (10elukey) 05Open>03Resolved [16:56:29] 10netops, 06Operations: pfws not on librenms - https://phabricator.wikimedia.org/T156381#2973000 (10faidon) pfw1 & pfw2 are members of each pair. They act as one control plane, so there is no reason (or way!) to add pfw1 and pfw2 separately. LibreNMS lists pfw-eqiad as "pfw1-eqiad" as before I fixed it there w... [18:18:48] cp1008 upgraded to openssl 1.1.0d and nginx restarted, ssllabs looks fine to me: https://www.ssllabs.com/ssltest/analyze.html?d=pinkunicorn.wikimedia.org&s=208.80.154.42 [18:19:15] should be good to rollout on production varnish, after exposing it for an hour to a single test host or so [18:19:21] openssl 1.0.2k coming tomorrow [18:19:25] moritzm: thanks [18:19:37] moritzm: I assume we were able to keep our existing patchwork? [18:20:03] yeah, your 8k patch is still around, no changes needed to be updated [18:20:09] yeah, your 8k patch is still around, no patches needed to be updated [18:20:12] ok cool, thanks again! [18:20:53] BTW; we can expect TLS 1.3 in openssl by April: https://mta.openssl.org/pipermail/openssl-announce/2017-January/000090.html [18:21:12] and since 1.1.1 will be API-compatible to 1.1.0 we can even easily upgrade [18:21:25] without having to mess with software upgrades like we did for nginx [18:21:39] awesome [19:33:09] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2973799 (10GWicke) [21:51:10] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links, 13Patch-For-Review: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2974262 (10Fjalapeno) @JMinor @JoeWalsh the fix for this is... [22:39:26] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2974386 (10GWicke) [22:39:55] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2781285 (10GWicke) [22:53:03] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2974435 (10GWicke) @Tgr and I discussed some more details in the office today. - General syntax: We both see general consensus around using query strings... [23:40:56] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2974687 (10RobH) [23:40:58] 10Traffic, 06Operations, 13Patch-For-Review: convert archiva to use Letsencrypt for SSL cert (deadline 2017-05-08) - https://phabricator.wikimedia.org/T154942#2974685 (10RobH) 05Open>03Resolved Conversion done, serving the new LE cert. I've removed the old certificate/key off the host, and out of puppe... [23:41:09] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10RobH)