[06:43:37] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Monitor Certificate Transparency (CT) logs - https://phabricator.wikimedia.org/T155807#2960473 (10faidon) I prepared packages for certspotter and filed a Debian ITP to maintain the package upstream (I'll wait for 0.3 to upload). I also built it (and its... [08:29:50] 10netops, 06Operations, 10ops-codfw: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2960554 (10faidon) [13:40:50] 10netops, 10DBA, 06Labs, 06Operations: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2961118 (10Marostegui) [15:09:00] 10Traffic, 06Operations, 06Performance-Team: Support brotli compression - https://phabricator.wikimedia.org/T137979#2385879 (10Eustaz) FYI: SDCH is on its way to [[ https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/nQl0ORHy7sw | unshippment ]]. [15:24:38] 10netops, 10DBA, 06Operations: Switchover s1 master db1057 -> db1052 - https://phabricator.wikimedia.org/T156008#2961450 (10Marostegui) [15:24:47] 10netops, 10DBA, 06Labs, 06Operations: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2961118 (10Marostegui) [15:27:39] Hey folks. I have some researchers who are very interested in looking studying our traffic and helping us develop new strategies for DDoS resilience. [15:27:51] Anyone likely to be interested in working with them? [15:27:58] I'd love to make some introductions. [15:29:46] halfak: you probably want to ping bblack directly when he's about :) [15:30:17] Thanks chasemp. I'll stick around for a while to see if I can catch him. Is he in UTC-8? [15:30:54] UTC -6 I believe (same as us) [15:31:55] Gotcha. Thanks. [15:32:09] UTC-6 represent! :D [15:38:32] halfak: we mostly work async, but put them in touch with ema and I if they have any bright ideas :) [15:39:20] bblack, ok email coming soon. [15:44:32] 10netops, 10DBA, 06Labs, 06Operations: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2961547 (10Marostegui) [17:09:23] 10netops, 10Analytics, 06Operations, 13Patch-For-Review: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2944637 (10Nuria) ping @elukey is this a duplicate [17:09:34] 10netops, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2961934 (10Nuria) [17:15:10] 10Traffic, 06Operations, 10Pybal, 15User-Joe: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#2961957 (10Joe) [17:24:19] 10netops, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Open temporary access from analytics vlan to new-labsdb one - https://phabricator.wikimedia.org/T155487#2961977 (10elukey) @Nuria no sorry this was probably the good one, I commented in https://phabricator.wikimedia.org/T155658... Sorry @JAlle... [17:26:11] 10Traffic, 06Operations: Enable Service in Asia Cache DC - https://phabricator.wikimedia.org/T156026#2961980 (10BBlack) [17:26:18] 10Traffic, 06Operations: Configuration for Asia Cache DC hosts - https://phabricator.wikimedia.org/T156027#2961994 (10BBlack) [17:26:22] 10Traffic, 06Operations: Name Asia Cache DC site - https://phabricator.wikimedia.org/T156028#2962007 (10BBlack) [17:26:24] 10Traffic, 06Operations: Select location for Asia Cache DC - https://phabricator.wikimedia.org/T156029#2962020 (10BBlack) [17:26:29] 10Traffic, 06Operations: Turn up network links for Asia Cache DC - https://phabricator.wikimedia.org/T156031#2962044 (10BBlack) [17:26:32] 10Traffic, 06Operations: Hardware installation for Asia Cache DC - https://phabricator.wikimedia.org/T156032#2962057 (10BBlack) [17:26:36] 10Traffic, 06Operations: Hardware purchasing for Asia Cache DC - https://phabricator.wikimedia.org/T156033#2962070 (10BBlack) [17:27:02] 10Traffic, 06Operations: Configuration for Asia Cache DC hosts - https://phabricator.wikimedia.org/T156027#2962084 (10BBlack) [17:27:04] 10Traffic, 06Operations: Enable Service in Asia Cache DC - https://phabricator.wikimedia.org/T156026#2962083 (10BBlack) [17:27:27] 10Traffic, 06Operations: Name Asia Cache DC site - https://phabricator.wikimedia.org/T156028#2962086 (10BBlack) [17:27:29] 10Traffic, 06Operations: Configuration for Asia Cache DC hosts - https://phabricator.wikimedia.org/T156027#2961994 (10BBlack) [17:27:52] 10Traffic, 06Operations: Select location for Asia Cache DC - https://phabricator.wikimedia.org/T156029#2962088 (10BBlack) [17:27:55] 10Traffic, 06Operations: Name Asia Cache DC site - https://phabricator.wikimedia.org/T156028#2962007 (10BBlack) [17:30:36] 10Traffic, 06Operations: Select location for Asia Cache DC - https://phabricator.wikimedia.org/T156029#2962108 (10BBlack) [17:31:33] 10Traffic, 06Operations: Select location for Asia Cache DC - https://phabricator.wikimedia.org/T156029#2962020 (10BBlack) [17:31:35] 10Traffic, 06Operations: Turn up network links for Asia Cache DC - https://phabricator.wikimedia.org/T156031#2962111 (10BBlack) [17:33:39] 10Traffic, 06Operations: Enable Service in Asia Cache DC - https://phabricator.wikimedia.org/T156026#2962115 (10BBlack) [17:33:41] 10Traffic, 06Operations: Hardware installation for Asia Cache DC - https://phabricator.wikimedia.org/T156032#2962116 (10BBlack) [17:33:54] 10Traffic, 06Operations: Hardware purchasing for Asia Cache DC - https://phabricator.wikimedia.org/T156033#2962118 (10BBlack) [17:33:56] 10Traffic, 06Operations: Hardware installation for Asia Cache DC - https://phabricator.wikimedia.org/T156032#2962057 (10BBlack) [17:35:48] 10Traffic, 06Operations: Turn up network links for Asia Cache DC - https://phabricator.wikimedia.org/T156031#2962126 (10BBlack) [17:35:50] 10Traffic, 06Operations: Enable Service in Asia Cache DC - https://phabricator.wikimedia.org/T156026#2962125 (10BBlack) [17:36:36] 10Traffic, 06Operations: Hardware purchasing for Asia Cache DC - https://phabricator.wikimedia.org/T156033#2962132 (10BBlack) [21:27:04] bblack: Where are the 4xx (specifically 404s) graphed/logged? [21:34:39] ostriches: it kinda depends, there's a few places that are easier for different purposes [21:34:50] https://grafana.wikimedia.org/dashboard/db/varnish-aggregate-client-status-codes [21:35:13] ^ is the one I use the most, which lets you break it down by-status-code (all 4xx, or specifically 404), and by datacenter and/or cluster [21:35:40] Main question is: I'm wanting to drop the old wiki.phtml entry point from our docroots. Apache *should* already be rewriting these (of which there aren't many...it *is* ancient) [21:35:48] Mostly looking for "oh shit, 404s spiked" [21:36:00] that one, then [21:36:19] you can filter down to only "text" cluster to only be looking at mediawiki+services (instead of maps, misc, and thumbnails, etc) [21:36:25] Perfect, this should do it [21:36:26] Thx [21:36:44] cache_type: text, status_type: 4 [21:55:34] Graph confirmed what I hoped: apache already rewrites this and removing the file from the docroot had no impact on 404s :) [22:01:05] cool :)