[08:47:42] 10Traffic, 10Monitoring, 06Operations, 13Patch-For-Review: diamond crashing on hosts using systemd-timesyncd - https://phabricator.wikimedia.org/T157794#3094776 (10MoritzMuehlenhoff) 05Open>03Resolved Closing, the collector isn't applied to systems using timesyncd any longer (and this is fixed on the c... [09:25:10] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3094868 (10elukey) Set `innodb_buffer_pool_size = 2048M` to see if helps. I checked `SHOW ENGINE INNODB STATUS` and some data is relevan... [09:32:33] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3094891 (10elukey) @JoeWalsh: Hi! I am wondering if the 100% -> 10% reduction in the sample rate has been released or not, since I am cu... [11:15:20] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3095154 (10JoeWalsh) @elukey the reduction to 10% will be in the next app version, 5.4. Currently it's scheduled to be released at the e... [12:44:39] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 2 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3095397 (10ema) >>! In T154954#3092101, @AndyRussG wrote: > The patch might purge 3000 or more URLs for each banner save... [13:05:14] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095504 (10Gilles) [13:10:31] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095512 (10Gilles) @BBlack have there been any known networking incidents during that period? 2017-03-09 05:00 -> 2017-03-10 13:00 The higher TTFB has been confirmed with diffe... [13:33:26] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095584 (10BBlack) Yes, the anomalies you're describing here were the result of a DDoS attack against us, and our mitigations to reduce user impact. The incident doc is private... [13:36:13] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095601 (10Gilles) @MoritzMuehlenhoff gave me the link. Is there a page we can watch to stay up to date on those? Since the depooling/repooling steps didn't show up in the SAL.... [13:39:11] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095604 (10BBlack) Presently, we don't have any sort of "live" feed on security incidents other than what you've already mentioned. For some classes of incident, such a thing w... [13:44:13] 10Traffic, 06Operations, 06Performance-Team: What happened 2017-03-09 04:00 - 06:00 UTC - https://phabricator.wikimedia.org/T160109#3095606 (10Gilles) I meant something private. The incident page on officewiki looks orphaned. If there was a page where those are listed, I could go there like I go to the SAL w... [16:45:13] 10netops, 10Continuous-Integration-Infrastructure, 06Labs, 06Operations: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3096072 (10EddieGP) p:05Triage>03Normal [17:04:56] 10netops, 10Continuous-Integration-Infrastructure, 06Labs, 06Operations: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3096149 (10hashar) 05Open>03Resolved a:03hashar Must have been a transient issue. Seems the bandwidth is... [22:28:20] 10Traffic, 06Labs, 06Operations, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3096884 (10Ciencia_Al_Poder)