[06:18:45] 10Traffic, 06Operations: convert dumps to use Letsencrypt for SSL cert (deadline: 2017-04-26) - https://phabricator.wikimedia.org/T154940#2944273 (10Dzahn) a:03Dzahn [07:09:09] 10Traffic, 10MediaWiki-API, 06Operations: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#2944333 (10Tgr) Yeah, [[https://github.com/wikimedia/operations-puppet/blob/fb4d97d7dab38124a0a32a0a6c728f033fac0abf/modules/varnish/templates/text-common.inc.vcl.e... [10:59:12] 10netops, 06Analytics-Kanban, 06Operations: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2944653 (10elukey) p:05Triage>03Low a:05MoritzMuehlenhoff>03None [11:08:12] 10netops, 06Analytics-Kanban, 06Operations: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2944677 (10JAllemandou) [11:25:48] 10Traffic, 10DNS, 06Operations, 07Beta-Cluster-reproducible: Ferm/DNS library weirdness on deployment-mediawiki boxes - https://phabricator.wikimedia.org/T153468#2881386 (10MoritzMuehlenhoff) Could report/send the patch upstream? [13:18:19] 10Traffic, 10DNS, 06Operations, 07Beta-Cluster-reproducible, 07Upstream: Ferm/DNS library weirdness on deployment-mediawiki boxes - https://phabricator.wikimedia.org/T153468#2944820 (10Krenair) Yeah. The real bug seems to be in Net::DNS, the trailing full stop config change + patch above for ferm is just... [13:23:45] 10Traffic, 10DNS, 06Operations, 07Beta-Cluster-reproducible, 07Upstream: Ferm/DNS library weirdness on deployment-mediawiki boxes - https://phabricator.wikimedia.org/T153468#2944830 (10Krenair) > You have successfully confirmed your subscription request to the mailing list net-dns-users, however final ap... [16:19:33] 07HTTPS, 10Traffic, 06Operations, 10Wikidata: wikiba.se should use HTTPS - https://phabricator.wikimedia.org/T155359#2945239 (10Izno) [16:22:01] 07HTTPS, 10Traffic, 06Operations, 10Wikidata: wikiba.se should use HTTPS - https://phabricator.wikimedia.org/T155359#2941299 (10Dzahn) wikiba.se is not owned by WMF and doesn't point to WMF DNS servers and the IP it points to is also not under our control. Therefore i don't really see how this is an Opera... [16:23:49] 07HTTPS, 10Traffic, 06Operations, 10Wikidata: wikiba.se should use HTTPS - https://phabricator.wikimedia.org/T155359#2945287 (10Dzahn) 05Open>03stalled [16:38:36] 07HTTPS, 10Traffic, 10Wikidata: wikiba.se should use HTTPS - https://phabricator.wikimedia.org/T155359#2945343 (10abian) >>! In T155359#2945259, @Dzahn wrote: > wikiba.se is not owned by WMF and doesn't point to WMF DNS servers and the IP it points to is also not under our control. > > Therefore i don't rea... [16:51:53] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2945413 (10Fjalapeno) [18:09:27] 10Traffic, 10MediaWiki-API, 06Operations: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#2945672 (10Anomie) No, because the module might not know in the first place. Look at the crazy things that cause trouble for {T127233}, it's basically the same thin... [18:29:02] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2945767 (10Andrew) [18:29:04] 10Traffic, 06Operations, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2945766 (10Andrew) 05Open>03Resolved [18:53:29] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2945933 (10Dzahn) [19:03:04] 10Traffic, 06Operations, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2946029 (10RobH) [19:16:08] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2946097 (10RobH) [19:16:52] 10Traffic, 06Operations: convert stream.wikimedia.org from GS to LE certificate - https://phabricator.wikimedia.org/T155524#2946101 (10RobH) [19:52:12] codfw backends refilling: https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats?panelId=21&fullscreen&var-datasource=codfw%20prometheus%2Fops&var-cluster=cache_upload&var-layer=backend&from=now-12d&to=now [19:54:38] ema neat, what's was the catalyst for that? a specific changeset or ? [19:55:05] chasemp: we re-routed ulsfo through codfw [19:55:05] codfw was depooled due to network issues [19:55:20] so yeah now users aren't on it yet, but ulsfo's misses are passing through it and helping refill it [19:55:44] it's so neat to have object count graphs now [19:55:53] yeah, godog++ [19:55:54] before we had to guess/infer/hope :) [19:56:16] or dstat --varnishstat on all hosts, which is something but not great really [19:57:29] yeah this is pretty awesome afa stats (cycling through datasources and layers for fun) [20:01:20] quite interesting to see the n_objects drops due to weekly backend restarts too [20:01:27] https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats?panelId=21&fullscreen&var-datasource=eqiad%20prometheus%2Fops&var-cluster=cache_maps&var-layer=backend&from=now-7d&to=now [20:16:41] bblack: should we try switching to systemd-timesyncd in ulsfo? https://gerrit.wikimedia.org/r/#/c/330865/ [20:22:00] later perhaps, lunch first :) [20:46:39] ema: neat! glad it is useful [20:47:34] more to come too, it is relatively easy to add new metrics to be aggregated top-level across DCs [20:47:51] I'm preparing the prometheus presentation and will need to update wikitech in the process too [20:49:13] anyways if you are interested in adding some in the meantime, it is essentially changing two files modules/role/files/prometheus/rules_ops.conf for the rules themselves and the "filters" for global metrics to be picked up in modules/role/manifests/prometheus/global.pp [22:27:16] 10Traffic, 06Operations, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Links: Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org - https://phabricator.wikimedia.org/T155504#2947316 (10JMinor) p:05Triage>03Normal