[00:02:33] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2974748 (10GWicke) [00:48:31] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2975008 (10RobH) [00:48:33] 10Traffic, 06Operations: convert stream.wikimedia.org from GS to LE certificate - https://phabricator.wikimedia.org/T155524#2975007 (10RobH) 05Open>03declined [00:48:46] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10RobH) [00:49:00] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10RobH) This likely shouldn't close yet, and we should add in mx/mail systems. [04:56:38] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2975266 (10Dzahn) @hashar which CI systems had SSL certs again please [08:14:54] ema: I've uploaded the new openssl packages to carbon, note you can't install these via debdeploy (but I think you'll do a full dist-upgrade anyway) [08:15:57] the initial 1.1 packages were built with the same source package name/openssl and since debdeploy forces strict versioning to the version specific in the YAML file, it stumbles over mismatching binary package versions of 1.1 and 1.0.2 [08:16:24] that's fixed the new packages, but the old 1.1.0c packages need to be upgraded via salt/apt-get first [08:46:42] 10Wikimedia-Apache-configuration: Add /zh-mo as an alias for wikipedia.org/w/index.php - https://phabricator.wikimedia.org/T49305#2975525 (10Liuxinyu970226) [09:32:03] moritzm: yeah I'm old school apt-get anyways [09:32:11] thanks! [09:32:30] duuude, apt ftw [09:37:05] 10netops, 06Labs, 06Operations: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2975634 (10faidon) asw-c2-eqiad was replaced yesterday (Jan 26 17:50 UTC) with one of our spares. Total downtime was approximately 30 minutes mostly due to the recabling effort but... [09:40:40] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2975637 (10hashar) @Dzahn I should have written down somewhere following our conversation from last week or so. For the CI we have the following domains all serving HTTP being force redirecte... [09:43:06] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2975639 (10hashar) [10:34:43] really nice reading - https://calendar.perfplanet.com/2016/http2-push-the-details/ [10:49:06] elukey: you might want to send it to ops or wikitech list ? :} [10:52:27] hashar: I found it in the performance ML while reviwing my backlog of emails to read :) [11:08:44] HTTP/2 push sounds interesting. Since MediaWiki knows the .css/.js etc resources associated with the HTML payload it has just sent [11:08:54] I guess that it can push the material [11:09:06] no clue how much havoc it is going to be to implement that [11:09:26] nor how caching will work (cause surely we want to push from the varnish cache, not from the mw app server) [11:54:50] the article is really nice since it highlights the major "road blocks" that the various strategies have [11:55:04] (or might incurr into) [11:55:54] really interesting problem to tackle, not sure how the perf team will approach it [12:09:51] 10netops, 06Labs, 06Operations: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2957555 (10fgiunchedi) >>! In T155875#2975634, @faidon wrote: > During the whole 30 minute window there was also an increased response time from the MediaWiki API, that cascaded in... [13:23:38] ema: bblack: I was looking at the vcl change "cache_misc req_handling: subpaths, cache policy, defaulting" https://gerrit.wikimedia.org/r/#/c/300581/ [13:23:59] does subpaths here mean we could do routing based on the path of the url so different paths ends up on different backend/ports? [13:24:07] hashar: yes [13:24:11] but [13:24:15] that is awesome! [13:24:28] :) [13:24:35] the idea in that commit is mostly to get thigns like that to be data-driven [13:24:42] I got a production use case if you get interested in testing for real at one point [13:24:54] the result today on varnish of applying that technique kind of sucks, because of the complexity of the ruby templating to generate code for it [13:25:14] integration.wikimedia.org is on misc-web cache with a backend apache that does some proxy to various services on the bo [13:25:14] x [13:25:18] but ultimately the fault there is not in having the cache code driven by declarative data (which is good!) [13:25:26] that is fairly isolated and could be a good validation [13:25:43] the fault ther is that VCL isn't a real language. if it were, we could just export the data and have native VCL consume it properly [13:26:10] in the future, ATS+Lua will consume the same data, but in a better way that sucks less [13:26:35] :-D [13:26:46] hashar: it's already in live use on cache_misc, for the eventstreams-vs-rcstream case [13:27:20] https://github.com/wikimedia/operations-puppet/blob/production/modules/role/manifests/cache/misc.pp#L252 [13:28:35] but wait [13:28:42] you even updated the inline comment / doc https://gerrit.wikimedia.org/r/#/c/300581/28/modules/role/manifests/cache/misc.pp !! [13:29:44] heh those docs look wrong, I think I lost a change to them somewhere in all the abandon->reclaim->rebase stuff [13:30:26] yeah [13:30:31] well anyways "subpaths - hash - If present, this is the only allowed key." is wrong. I moved away from having subpaths be the only key if present (where it would need a default subpath), to having the upper-level information be the default. [13:30:42] maybe :} But at least that explains the feature enough for me to get it [13:32:14] next steps on that little railroad are to move the code from cache_misc to our core shared VCL, and support multiple (per-dc) backend hostnames per director [13:32:22] and then use that data as the basis for our active/active support [13:35:31] (on a per-service level instead of per-cluster, which is all we could do today) [14:34:01] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2976419 (10Dzahn) @hashar thank you for this very detailed reply. Since everything is already behind varnish i think it will not be relevant in the context of this ticket then because of "2)"... [14:38:16] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2976449 (10hashar) I don't think we ever used self-signed certs for CI. Internal communications I can remember of are: * Varnish -> Apache on contint1001 (plain HTTP) * Apache on contint1001 t... [15:37:15] 10Traffic, 06Operations: Select or Acquire Address Space for Asia Cache DC - https://phabricator.wikimedia.org/T156256#2976687 (10BBlack) [15:58:08] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links, 13Patch-For-Review: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2976752 (10ema) @Fjalapeno @JMinor @JoeWalsh the issue shoul... [16:33:08] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2976836 (10Anomie) > The simplest solution would be to just move all thumb accesses to thumb.php (or an api module) I note thumb.php could probably use `... [16:41:17] 10Traffic, 06Commons, 06Operations, 10media-storage, 07Regression: Some JPGs are being served as text - https://phabricator.wikimedia.org/T148497#2976859 (10zhuyifei1999) 05Open>03Resolved Closing as resolved as it cannot be reproduced anymore. If the bug appears again feel free to reopen. [16:56:43] 10netops, 06Operations: pfws not on librenms - https://phabricator.wikimedia.org/T156381#2976889 (10faidon) 05Open>03Resolved a:03faidon I reenabled (actually removed and readded) pfw-codfw yesterday and I haven't seen any ill effects in ~24 hours, so resolving this. By the way, note that pfws are easily... [20:55:04] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links, 13Patch-For-Review: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2977751 (10Fjalapeno) @ema thanks for the update! We will ge... [20:56:32] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links, 13Patch-For-Review: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2977752 (10JoeWalsh) @ema thanks! it's working now [20:57:23] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links, 13Patch-For-Review: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2977755 (10Fjalapeno) 05Open>03Resolved @Ema - just chec...