[00:26:04] 10Traffic, 10DBA, 06Operations: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3188507 (10Dzahn) not sure if this should be tagged as traffic or not. please feel free to remove it. it just got auto-added because it copies tags when you create somethi... [02:30:46] 10Traffic, 06Operations, 10ops-codfw: baham (ns1) CPU-related issues - https://phabricator.wikimedia.org/T159870#3188751 (10BBlack) 05Open>03Resolved a:03Dzahn I'll close it for now. If we see more strange issues with super-low cpu freqs we can always search these up to correlate I guess. [09:23:19] 10netops, 06Operations: analytics hosts frequently tripping 'port utilization threshold' librenms alerts - https://phabricator.wikimedia.org/T133852#3189302 (10fgiunchedi) @ayounsi sounds good to me! I think for the longer period of time we can start with 3x (or 2x) the current 5min and see if that helps. Usua... [09:26:49] 10netops, 06Operations: analytics hosts frequently tripping 'port utilization threshold' librenms alerts - https://phabricator.wikimedia.org/T133852#3189317 (10faidon) Yup, sounds like a good idea. Thanks for taking care of that @ayounsi :) [09:26:51] 10Traffic, 06Operations, 10media-storage: swift-object-server 1.13.1: Wrong Content-Type returned on 304 Not Modified responses - https://phabricator.wikimedia.org/T162348#3189319 (10fgiunchedi) FWIW the swift 2.2.0 upgrade is complete (from T162609) [10:16:58] ema, bblack anything against me running the traffic-related steps of the switchdc? (disable puppet on eqiad/codfw, enable+run puppet on one and then the other) [10:17:50] volans: nope, please go ahead [10:17:59] ack, thanks! [12:20:44] 10netops, 06Operations: analytics hosts frequently tripping 'port utilization threshold' librenms alerts - https://phabricator.wikimedia.org/T133852#3189553 (10ayounsi) a:03ayounsi [13:02:54] 10netops, 06Operations, 10ops-esams: esams higher than usual temperature - https://phabricator.wikimedia.org/T162152#3189612 (10ayounsi) a:03ayounsi Looking at the graphs, now that the FPC has been replaced, the temperature is much more regular. Also the temperature variations seemed to match Amsterdam's w... [15:45:14] 10netops, 10Monitoring, 06Operations: nagios monitor transit/peering links and alert on low/high traffic - https://phabricator.wikimedia.org/T80273#3190110 (10ayounsi) a:03ayounsi Not sure if still relevant seeing how old the ticket is, but: High traffic: already being monitored by LibreNMS Low traffic... [16:08:45] 10netops, 06Operations: Filter outgoing BGP announcements on AS regex - https://phabricator.wikimedia.org/T83037#908416 (10ayounsi) Slightly different, it is possible to use the configuration statement "remove-private" to achieve a similar goal. It's not as strict as specifying the allowed AS# but might be a g... [16:13:51] 10netops, 10Monitoring, 06Operations: nagios monitor transit/peering links and alert on low/high traffic - https://phabricator.wikimedia.org/T80273#3190196 (10faidon) Yes, it does. Thanks for working on such an old task! For (1) of your list: - cr1-ulsfo.wikimedia.org:xe-0/0/3.98: this doesn't seem to be co... [16:39:24] 10netops, 06Operations: Filter outgoing BGP announcements on AS regex - https://phabricator.wikimedia.org/T83037#3190279 (10faidon) Yes, that was my intention as well. The other thing about this that I see on my notes is to `set as-path path 14907` either under the aggregate routes, or under defaults, plus pot... [18:15:36] 10Traffic, 10DBA, 06Operations: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10BBlack) Yeah, leave the traffic tag as we'll want to basically revert https://gerrit.wikimedia.org/r/#/c/348456/ once dbtree is ready for it. [18:19:02] 10Traffic, 10DNS, 06Operations, 06Services (next): icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3190756 (10BBlack) 05Open>03stalled @Gwicke yeah we should. Regardless, after we're done with the current codfw switchover/switchback, I thin... [18:24:18] 10Traffic, 06Operations: Implement Varnish-level rough ratelimiting - https://phabricator.wikimedia.org/T163233#3190763 (10BBlack) [18:24:32] 10Traffic, 06Operations: Implement Varnish-level rough ratelimiting - https://phabricator.wikimedia.org/T163233#3190776 (10BBlack) p:05Triage>03Normal [18:25:06] 10Traffic, 06Operations: Implement Varnish-level rough ratelimiting - https://phabricator.wikimedia.org/T163233#3190763 (10BBlack) [18:25:10] 10Traffic, 06Discovery, 06Maps, 06Operations, 03Interactive-Sprint: Rate-limit browsers without referers - https://phabricator.wikimedia.org/T154704#3190777 (10BBlack) [18:26:19] 10Traffic, 06Operations: Implement Varnish-level rough ratelimiting - https://phabricator.wikimedia.org/T163233#3190763 (10BBlack) [18:26:22] 10Traffic, 06Operations: Increase request limits for GETs to /api/rest_v1/ - https://phabricator.wikimedia.org/T118365#3190779 (10BBlack) [18:28:01] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Upgrade all cache clusters to Varnish 4 - https://phabricator.wikimedia.org/T131499#3190790 (10BBlack) [18:28:04] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#3190786 (10BBlack) 05Open>03Resolved Going back over some of the unchecked boxes at the top: # Ratelimiting and general VCL reload memleak issues can be investi... [18:30:11] 10Traffic, 06Operations: unix domain socket listening for varnish4 - https://phabricator.wikimedia.org/T138084#3190801 (10BBlack) 05Open>03Resolved a:03BBlack For now we've solved the pragmatic issues in other ways: some general nginx/varnish tuning, kernel TCP params tuning, and using 8x TCP sockets in... [19:52:38] 10Traffic, 06Operations, 06Community-Liaisons (Jul-Sep 2017): Communicate this security change to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3191313 (10Whatamidoing-WMF) [21:46:53] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191871 (10Krinkle) 05Open>03Resolved a:03Krinkle Landed in master for 1.28 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191909 (10demon) There's probably some extensions that need fixing here too. The Elastica library looks possibly... [21:55:25] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191910 (10demon) 05Resolved>03Open [22:05:21] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191940 (10demon) 05Open>03Resolved >>! In T140658#3191909, @demon wrote: > There's probably some extensions t... [22:05:34] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191943 (10demon) [22:59:16] 10Traffic, 06Operations: Select or Acquire Address Space for Asia Cache DC - https://phabricator.wikimedia.org/T156256#3192167 (10DFoy) @BBlack Is there an update for ETA for updating the zero whitelisting IPs? I will need several months of lead time before this goes live to get ~50 partners worldwide to upda... [23:16:03] 10Traffic, 06Operations: Select or Acquire Address Space for Asia Cache DC - https://phabricator.wikimedia.org/T156256#3192246 (10BBlack) We still have no real ETA on the IP addresses. We're attempting to acquire the address space from APNIC. They're (reasonably) requiring proof of our needs, which includes...