[02:00:59] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 7 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2093337 (10Legoktm) https://de.wikipedia.org/wiki... [05:05:09] 7HTTPS, 10Traffic, 6Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2093434 (10Tbayer) Update: Followed up again on March 4 and got yet another response from yet another person: It's actually possibly to use WP-CLI for replacements, but scripts can't proc... [09:52:25] 10Traffic, 6Operations, 13Patch-For-Review: Port varnishlog.py to new VSL API - https://phabricator.wikimedia.org/T128788#2093709 (10elukey) @ema: I discovered an interesting thing from https://www.varnish-cache.org/docs/trunk/reference/vsl.html: > Timestamp - Timing information > Contains timing informatio... [10:29:05] elukey: hi! [10:29:38] ema: o/ [10:29:46] elukey: yeah, that timestamping info LGTM [10:30:01] what was ReqEnd returning in v3? [10:31:12] don't really know... [10:31:14] :D [10:31:31] but we can figure it out very quickly [10:35:45] 10Traffic, 6Discovery, 10Maps, 6Operations, and 4 others: allow maps cluster Varnish cache purging - https://phabricator.wikimedia.org/T112836#2093761 (10Yurik) [10:36:00] 10Traffic, 6Discovery, 10Maps, 6Operations, and 2 others: Set up proper edge Varnish caching for maps cluster - https://phabricator.wikimedia.org/T109162#2093768 (10Yurik) [10:38:40] 7Varnish, 6Discovery, 10Maps, 10tilerator, 3Discovery-Maps-Sprint: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2093803 (10Yurik) [10:38:54] elukey: http://book.varnish-software.com/3.0/Getting_started.html#varnishlog-tag-examples [10:41:50] ema: https://github.com/wikimedia/varnishkafka/blob/master/varnishkafka.c#L778 [10:42:06] We are currently using stop-start timings [10:43:12] elukey: we seem to use it for TTFB as well https://github.com/wikimedia/varnishkafka/blob/master/varnishkafka.c#L975 [10:44:49] theoretically from https://www.varnish-cache.org/docs/trunk/reference/vsl.html it should be timing->process (time from start) [10:45:01] Process [10:45:02] Processing finished, ready to deliver the client response. [10:48:07] I checked and the field that I mentioned in Timestamp does increase from start to end [10:48:21] (not a good proof but at least it seems sound) [11:05:38] ema: https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishncsa/varnishncsa.c#L849 [11:06:01] If I am reading it correctly ncsa does the same thing [11:28:26] elukey: yep, %{Varnish:time_firstbyte}x is handled there indeed [11:29:16] using SLT_Timestamp, that is [11:42:09] 10Traffic, 6Operations: Images not showing up at Commons - https://phabricator.wikimedia.org/T128961#2093978 (10faidon) [12:52:50] ema: I have also discovered why in format_parse we have client/backend markers with different tags [12:53:30] when you have a HIT the marker is client, otherwise it is backend (so when a fetch is needed) [12:53:37] this is why we have to put both [12:53:54] and sometimes their related tags names can be different [12:56:15] (or something similar, I am still trying to understand the whole thing) [12:59:07] scratch everything, those are the different transactions [13:11:56] ema: https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishncsa/varnishncsa.c#L795 <---- [13:15:21] 10Traffic, 10ArchCom-RfC, 6Commons, 10MediaWiki-File-management, and 13 others: Use content hash based image / thumb URLs & define an official thumb API - https://phabricator.wikimedia.org/T66214#2094197 (10mark) [13:15:56] the new varnish ncsa does not consider backend tags at all.. [13:20:59] 10Traffic, 10ArchCom-RfC, 6Commons, 10MediaWiki-File-management, and 13 others: Use content hash based image / thumb URLs & define an official thumb API - https://phabricator.wikimedia.org/T66214#2094208 (10Gilles) What's the caching strategy for this API? Will it simply redirect/proxy to the canonical thu... [13:23:47] elukey: does not output backend tags even when a fetch is involved? [13:24:08] we don't really expect backend tags on a cache hit anyways [13:25:10] has anyone looked at that commons 404 at all? [13:27:18] we had a similar issue before, I think, where an ISP proxy was aliasing our domains inappropriately [13:27:51] (as in, they have some custom rules in their proxy that aliases all our domains to the text-lb address, and they included *.wikimedia.org in that list, even though DNS points names like upload or phab to different IPs) [13:35:23] bblack: this is an example of fetch (or I think so, please tell me if not) - https://dpaste.de/NOqE - ('c' in the end of each tag is an indicator of client marker) [13:35:44] the \u000 horrible outputs in the end is an encoding issue that I need to resolve [13:36:25] In this case I have a transaction (vxid) for the client request and one for the backend, containing other gats [13:36:25] is that from invoking varnishncsa? [13:36:28] *tags [13:36:50] nope the new varnish-kafka [13:37:19] ok so it's not getting backend tags [13:37:32] they may be completely separated and only related by some ID, I have no idea [13:37:43] atm it is not because I commented the step, but I can show you the whole transaction group [13:38:27] 10Traffic, 6Operations: Images not showing up at Commons - https://phabricator.wikimedia.org/T128961#2094218 (10BBlack) Note that `cp3043 frontend ... Error: 403, Requested target domain not allowed` is a legitimate error from the correct (upload) cluster defined here: https://github.com/wikimedia/operations-p... [13:39:26] bblack: this one is logging all the transactions related to a request: https://dpaste.de/Hucd [13:39:58] with the new logging API we can have transactions (client/backend/etc..) with related tags grouped, that is great [13:40:13] but my doubt is: do we need the backend tags? [13:40:53] because atm varnish kafka logs basically each trasaction, so one for client and one for backend fetch for example [13:43:38] the output doesn't make a lot of sense right now I know, but I'd need some guidance about what vk should really log [13:44:39] elukey: I think so long as we have response headers too, for current VK needs the client transaction is enough [13:45:01] elukey: but note we do log other stats from backend tags, via non-VK things [13:46:06] some of the things we tag in VK are response not request headers, because in the past request headers in VSL are raw ones, and don't reflect our modification of them in vcl_recv via e.g. 'set req.http.foo = "bar"' [13:46:18] so we end up having to reflect some things like that through response headers to get them in VK logs [13:46:24] (like X-Client-IP) [13:52:45] bblack: ahh ok! So the %{variable}o expansion should be pointed to the RespHeader tags in the examples that i showed to you right? For example, something like %{Content-Type@content_type}o [13:53:30] if so we have all the data in the client transaction indeed [13:54:00] elukey: my understanding is that in v3 they were using the same tag for client/backend stuff (eg: RxHeader) and that's why we needed to distinguish in varnishkafka between client and backend tags [13:54:23] with v4 though there is no ambiguity (eg: ReqHeader/BereqHeader) [13:54:46] ema: all right I've read Rx vs Tx in the docs that you sent me and it now makes sense [13:54:58] https://phabricator.wikimedia.org/T128788#2088709 [13:57:37] ema: yep but my doubt is.. do we need, for example, BereqURL if we can grab something like ReqURL: /?test=2743 ? [13:58:54] elukey: were we logging berequrl before? [13:58:56] well there could be a transformation in our VCL right. The client might send ReqURL: /?test=2743 and our VCL could rewrite it to BereqURL: /whatever/?test=2743 [13:59:20] yeah but we don't do that in frontends with berequrl, which is different than transforming req.url [14:00:27] so there's 3x actual VK format strings in use: [14:00:39] from git grep -w format modules/role/manifests/cache/kafka [14:00:49] eventlogging.pp: format => '%q %l %n %{%FT%T}t %{X-Client-IP@ip}o "%{User-agent}i"', [14:00:57] statsv.pp: $format = "%{fake_tag0@hostname?${::fqdn}}x %{%FT%T@dt}t %{X-Client-IP@ip}o %{@uri_path}U %{@uri_query}q %{User-Agent@user_agent}i" [14:01:10] format => "%{fake_tag0@hostname?${::fqdn}}x %{@sequence!num?0}n %{%FT%T@dt}t %{Varnish:time_firstbyte@time_firstbyte!num?0.0}x %{X-Client-IP@ip}o %{Varnish:handling@cache_status}x %{@http_status}s %{@response_size!num?0}b %{@http_method}m %{Host@uri_host}i %{@uri_path}U %{@uri_query}q %{Content-Type@content_type}o %{Referer@referer}i %{X-Forwarded-For@x_forwarded_for}i %{Use [14:01:16] r-Agent@user_agent}i %{Accept-Language@accept_language}i %{X-Analytics@x_analytics}o %{Range@range}i %{X-Cache@x_cache}o", [14:01:19] (that last one is webrequest) [14:01:59] (brb sorry, doorbell rang) [14:02:45] I don't think any of those have to do with backend tags [14:03:35] we don't necessarily have to support all possible future things, just what we have today, which I think is all frontend/client-facing req+resp headers + metadata [14:04:39] it's only the other python stats scripts (like varnishrls, varnishxcps, etc) that actually look at some backend fields, I think [14:04:42] even then, not all of them [14:22:13] bblack: got it thanks! So I'll proceed with whitelisting only "client" tags without any backend one for the first version ok vk. ema does it sounds ok for you? [14:23:00] the other python stats scripts might need to look into multiple transactions, but it shouldn't be a problem to code it in python [14:25:13] ok so... back on the commons 403 thing [14:25:24] I started digging in oxygen logs and found a lot of unexplained 403s [14:26:04] the specific case I've been narrowing in on is frontend "int" (internal 403, generated by VCL), for a GET request to a legitimate hostname [14:27:05] e.g.: [14:27:05] "http_method": "GET", [14:27:05] "uri_host": "upload.wikimedia.org", [14:27:05] "uri_path": "/wikipedia/commons/thumb/3/3f/Typy_panensk%C3%BDch_blan.png/180px-Typy_panensk%C3%BDch_blan.png", [14:27:08] "uri_query": "", [14:27:11] "x_cache": "cp3045 frontend int(0)" [14:27:22] ^ the above seems to make no sense. nothing in our VCL should be 403-ing that request [14:27:48] and I've found one source of the problem [14:28:32] oxygen's "uri_host" is after varnish breaks things down for stats purposes, but I've found matching live requests that look similar in varnishlog on frontends, and they show: [14:28:36] 43 RxHeader c Host: upload.wikimedia.org:80 [14:28:53] I think 'req.http.host' in VCL still has the :80, and that's screwing up our regex matches on hostname [14:29:09] (or non-regex matches for that matter) [14:30:21] of course those :80 requests actually are direct to port 80 (http), so they should be 301 anyways, but that's neither here nor there [14:31:20] uh [14:31:43] and tbh, I think that could be happening from our own rewrite_proxy_urls too [14:31:57] if(req.url ~ "(?i)^https?://[^/]") { set req.http.Host = regsub(req.url, "(?i)^https?://([^/]+).*$", "\1"); set req.url = regsub(req.url, "(?i)^https?://[^/]+", ""); [14:32:52] not that any client should be sending us proxy-style requests, but most of these I've noted come from ISP proxy IPs. They may be hoping that hitting us with "GET http://upload.wikimedia.org:80/foo HTTP/1.1" will bypass HTTPS redirects [14:33:00] (which it kinda does, but it makes them 403s heh) [14:33:06] :) [14:33:26] if (req.http.host != "<%= @vcl_config.fetch('upload_domain') %>") { [14:33:30] yeah that [14:33:45] (which should probably be case-insensitive anyways, but that's neither here nor there) [14:34:36] so probably 1) We should strip :port from req.http.host somewhere in shared VCL that makes sense, before other things try to parse/match it [14:34:48] 2) We should fix rewrite_proxy_urls to not copy a port to the host field either [14:34:59] 3) Maybe we should case-normalize req.http.host early too [14:35:14] to avoid case-insensitive matchings everywhere, agreed [14:36:19] sicne rewrite_proxy_urls is "special", probably just fix the :port -copying there, and then do case-normal and port-stripping after that step for all requests, before anything else gets a chance to look at the hostname [14:36:50] hmmm rewrite_proxy_urls is quite late compared to some other bits, like the https stuff [14:44:42] the ordering of some of the VCL seems so clearly not-quite-right now (that it's been broken up and refactored a little differently and more-clearly) [14:44:54] before it was such a jumble it was hard to see these things heh [14:45:11] ema: can you check out https://gerrit.wikimedia.org/r/275474 [14:45:32] bblack: looking [14:50:30] bblack: we sometimes use req.http.Host and some other times req.http.host [14:51:28] should we try to stick to one of the two? [14:52:55] probably. I see did both in one line just now too lol [14:53:17] :) [14:53:52] Host is on 59 lines and host is on 29 lines in terms of popularity [14:54:02] Host it is then [14:54:05] Host: is how I usually think of it canonically in my head [14:54:25] but on the other hand, if we're just going with a rule of "varnish doesn't care, but let's normalize all our req.http.Foo", lowercase makes more sense [14:54:44] bikeshedding++ [14:55:23] I can at least make this one patch consistent. it will be some followup patch if we want to normalize all the refs anyways. [14:55:33] sounds good [14:57:04] I guess bikeshedding is double-appropriate here, since WP says: [14:57:09] "The term bike-shedding or the bike-shed effect was coined as a metaphor to illuminate the law of triviality; it was popularised in the Berkeley Software Distribution community by the Danish computer developer Poul-Henning Kamp in the mid-1990s[3] and has spread from there to the whole software industry." [14:57:21] :) [14:58:24] stupid question of the day (SQOD): how about URLs such as https://ema:password@upload.wikimedia.org/test [14:58:59] the regexp would match 'ema:' right? [15:01:31] I suspect the whole proxy comes from rewrite_proxy_urls anyways. probably varnish breaks it down anyways for the usual req.http.host [15:01:49] well what am I thinking, there would never be user:pass in normal req.http.host [15:02:03] mmh but we set req.http.Host ourselves a few lines above [15:02:08] * bblack curses HTTP standards for not just using putting the hostname in the request line in the first place [15:02:28] if(req.url ~ "(?i)^https?://[^/]") { set req.http.Host = regsub(req.url, "(?i)^https?://([^/]+).*$", "\1"); [15:02:35] I guess what I mean is, only the proxy decoder would do that [15:02:38] we could fix it there, too [15:03:07] hmmm [15:04:15] unlike the user:pass@ part, though, :port actually is legal in a host-header, although it's not clear to me if varnish strips that for req.http.Host normally [15:04:30] so IMHO we only have to pre-strip user:pass@ in the proxy case [15:06:40] * ema can't find "the proxy case" [15:10:51] bblack: is it just rewrite_proxy_urls or do we have other VCL snippets dealing with that? [15:11:01] no, just rewrite_proxy_urls [15:11:21] because we've run into those before, due to buggy/horrible clients or clients' proxies [15:11:50] anyways, PS3 should fix all of that [15:15:36] bblack: LGTM! [15:29:08] bblack: uh, syntax error [15:31:06] it's the single quotes I guess? [15:31:35] yup [15:54:41] bblack: a little too late to catch the syntax error, but still: https://phabricator.wikimedia.org/P2716 [15:56:08] we can add it to the tests some sunny day [16:21:01] 10Traffic, 6Operations: Images not showing up at Commons - https://phabricator.wikimedia.org/T128961#2094653 (10BBlack) We've fixed a some subtle issues on our end with the de-coding of proxy-style URIs which included a port number, meaning the client request from our perspective was of the form `GET https://u... [16:22:58] ema: I honestly haven't played with your VTC stuff at all, since I usually do my work from a host that doesn't have varnish and whatnot (which will need to change!). But is there any chance the basic vtc stuff will work fine on v3 anyways, in which case we could push at least some of the tests ahead of the varnish4 patch [16:23:09] ema: at least, the basic "does it compile at all" test would be a huge start [16:23:28] and then we can start asking to hook it up to jenkins too [16:23:31] bblack: yes, the problem is that v3 does not support regexp matching :( [16:23:42] so all tests using the ~ operator would fail [16:23:48] ok [16:24:06] but tests like the one above work just fine [16:24:16] (tried on cp1008) [16:24:35] eventually we could expand this to test a lot of our applayer contracts, which are poorly-documented [16:24:43] by mocking the applayer behaviors we expect, etc [16:26:03] bblack: I can try to pick all tests that don't necessarily require regexp matching and add them to v3 [16:26:23] it doesn't make much sense to mix all that in the VCL porting patch anyways [16:26:51] yeah [16:28:56] bblack: in other news, I came up with a quick and dirty varnishkafka in python on cp1008:/home/ema/vkncsa.py [16:29:10] I haven't added kafka output yet though [16:29:34] it just takes varnishncsa's output, converts to JSON and adds the missing fields (hostname and sequence) [16:29:55] interestingly, by using the json module CPU usage is unacceptable (double digit) [16:30:04] heh [16:30:10] with cjson, instead, it is comparable to the other python varnish modules [16:30:24] ~ 5% with lots of requests (all the PURGEs) [16:30:26] ok, that kinda makes sense [16:30:44] oh yeah, and I've installed python-cjson on cp1008, I hope it's fine [16:30:52] yeah that's fine [16:31:16] we'll want to support something similar to varnishkafka's "format" so that we can configure fields to grab and encode to the json [16:31:38] yeah that's supported somehow with a little parser [16:31:48] probably buggy, but good enough for a PoC [16:32:53] yeah [16:33:04] not having more C code to main would be a nice win here! [16:33:24] now I would need to throw all that stuff to a test kafka instance [16:33:28] does https://github.com/edenhill/rdkafka-python work for the output? [16:34:31] bblack: I still have to try, but I don't see why it shouldn't [16:35:07] I'm not sure how to proceed for testing kafka output though [16:35:14] me either :) [16:35:17] :) [16:35:28] ottomata may have stuff to do that he's used before though [16:35:37] I will ask him [16:40:55] 7HTTPS, 10Traffic, 6Labs, 6Operations, 10Tool-Labs: Make Magnus tools on tools.wmflabs.org work in HTTPS - https://phabricator.wikimedia.org/T102457#2094745 (10Magnus) Just FYI, treeviews is now completely https (the last remaining http element was the tree image...) https://tools.wmflabs.org/glamtools/t... [16:42:35] 7HTTPS, 10Traffic, 6Labs, 6Operations, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2073537 (10Magnus) @Dzahn: Does "using http" mean "available over http", or "do not work on https"? [17:10:13] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#2094977 (10BBlack) [17:10:15] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2094978 (10BBlack) [17:10:18] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Refactor VCL for applayer datacenter-switching - https://phabricator.wikimedia.org/T127484#2094974 (10BBlack) 5Open>3Resolved a:3BBlack This works now, and is controlled by per-cluster hieradata in the `apps` s... [17:14:51] 10Traffic, 10Analytics, 10Datasets-General-or-Unknown, 6Operations: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2094989 (10ArielGlenn) [17:15:31] 10Traffic, 10Analytics, 10Analytics-Cluster, 6Operations: Enable Kafka native TLS in 0.9 and secure the kafka traffic with it - https://phabricator.wikimedia.org/T121561#1881737 (10Milimetric) p:5Triage>3Normal [17:21:25] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL source-DC switching via confd - https://phabricator.wikimedia.org/T127482#2095026 (10BBlack) [17:22:07] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#2095029 (10BBlack) [17:24:45] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2095039 (10BBlack) Removing the 2x confd-related blocker tasks: they'll still be open tasks tagged for #codfw-roll... [17:24:55] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#2095042 (10BBlack) [17:24:57] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL source-DC switching via confd - https://phabricator.wikimedia.org/T127482#2095043 (10BBlack) [17:25:00] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2095041 (10BBlack) [17:25:08] 10Traffic, 6Operations, 5codfw-rollout: Enable VCL source-DC switching via confd - https://phabricator.wikimedia.org/T127482#2044847 (10BBlack) [17:25:35] 10Traffic, 6Operations, 5codfw-rollout: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#2044954 (10BBlack) [17:27:23] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2095054 (10BBlack) Overall status update: work is complete at the configuration level to support the necessary swi... [17:29:33] 7HTTPS, 10Traffic, 6Labs, 6Operations, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2095064 (10scfc) @Magnus: In this context, it means that a request came in with `http` and was not redirected or failed. This could be by someone typing in th... [17:29:53] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Switch ulsfo to backend to codfw rather than eqiad - https://phabricator.wikimedia.org/T127492#2095068 (10BBlack) This is ready to test. Will shoot for early Tuesday (tomorrow). Should not impact anything else. [18:34:52] 10Traffic, 10ArchCom-RfC, 6Commons, 10MediaWiki-File-management, and 13 others: Use content hash based image / thumb URLs & define an official thumb API - https://phabricator.wikimedia.org/T66214#2095376 (10GWicke) > What's the caching strategy for this API? Will it simply redirect/proxy The most efficien... [18:38:24] 10Traffic, 6Operations, 13Patch-For-Review: Port varnishlog.py to new VSL API - https://phabricator.wikimedia.org/T128788#2095398 (10ori) [18:55:54] 10Traffic, 6Operations, 6Performance-Team: Segment Navigation Timing data by continent - https://phabricator.wikimedia.org/T128709#2083445 (10ori) p:5Triage>3High [19:15:26] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 7 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2095706 (10ori) >>! In T124356#2090639, @Jdlrobso... [20:08:02] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 7 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2096022 (10Jdlrobson) @ori I don't appreciate yo... [21:18:09] 10Traffic, 10Monitoring, 6Operations, 10Scap3 (scap3-adoption): Deploy libreNMS with scap3 - https://phabricator.wikimedia.org/T129136#2096567 (10greg) Added #monitoring and #traffic assuming that hits the right people who care about libreNMS. [21:29:07] 10Traffic, 10Monitoring, 6Operations, 10Scap3 (scap3-adoption): Deploy libreNMS with scap3 - https://phabricator.wikimedia.org/T129136#2096721 (10thcipriani)