[08:41:10] 10Traffic, 10Operations, 10ops-eqiad: cp1068 memory correctable errors - https://phabricator.wikimedia.org/T194757#4207584 (10ema) p:05Triage>03Normal [08:41:36] 10Traffic, 10Cloud-Services, 10Operations, 10cloud-services-team, and 3 others: Deprecate `base::service_unit` in puppet - https://phabricator.wikimedia.org/T194724#4209506 (10ema) p:05Triage>03Normal [09:10:13] 10Traffic, 10Operations, 10Patch-For-Review: Unconditional return(deliver) in vcl_hit - https://phabricator.wikimedia.org/T192368#4209526 (10ema) [09:36:31] https://www.fastly.com/blog/headers-we-dont-want [09:39:32] so they want to stop sending x-aspnet-version?? [09:40:49] looks like, we could s#HHVM/3.18.6-dev#ASP.NET# just for the lulz instead [09:41:12] or x-powered-by: coldfusion [09:41:29] s/HHVM/IMPS [09:41:31] O:) [09:41:41] (Infinite Monkey Protocol Suite) [09:43:59] hmm with our request numbers.. it would be interesting to check the amount of traffic saved by cleaning "vanity headers" [09:47:18] some good candidates for removal: x-powered-by, x-varnish, (possibly) via, (possibly) server [09:48:41] us networkers don't like it when you remove traffic! [09:49:00] x-analytics [09:49:46] we actually pay for our edge traffic, right? [09:51:35] in some cases [10:18:44] 10Traffic, 10Operations, 10media-storage: Remove unnecessary response headers - https://phabricator.wikimedia.org/T194814#4209672 (10ema) [10:18:55] 10Traffic, 10Operations, 10media-storage: Remove unnecessary response headers - https://phabricator.wikimedia.org/T194814#4209683 (10ema) p:05Triage>03Normal [10:22:54] 10Traffic, 10Operations, 10media-storage: Remove unnecessary response headers - https://phabricator.wikimedia.org/T194814#4209672 (10MoritzMuehlenhoff) We could also simply avoid X-Powered-By at the source; our PHP configs already use "expose_php=off" and for HHVM per https://github.com/facebook/hhvm/issues/... [10:31:43] 10Traffic, 10Operations, 10media-storage: Remove unnecessary response headers - https://phabricator.wikimedia.org/T194814#4209672 (10Joe) The `X-Powered-By` part is actually useful for us in order to discern the source of rendering of a page - be it hhvm or php. We will use it during the HHVM => PHP7 migrat... [10:32:11] 10Traffic, 10Operations, 10media-storage: Remove unnecessary response headers - https://phabricator.wikimedia.org/T194814#4209672 (10fgiunchedi) Ditto for some #thumbor headers: ``` thumbor-engine: wikimedia_thumbor.engine.imagemagick thumbor-processing-time: 413 thumbor-processing-utime: 316 thumbor-reques... [10:34:10] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090#4209746 (10ayounsi) While preparing the firewall rule for Dallas I discovered a limitation not accounted for previously. The rule that says "if ping to VIPs, then redirect to... [12:41:43] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#4210035 (10Aklapper) >>! In T174342#3790202, @Mholloway wrote: > I've reached out to Partnerships about getting in touch with Maroc and INWI for IP range updates. @Mholloway:... [12:42:49] 10netops, 10Cloud-Services, 10Operations: Allocate public v4 IPs for Neutron setup in eqiad - https://phabricator.wikimedia.org/T193496#4210037 (10faidon) The /25 -> /24 renumbering seems fairly straightforward, but given a) IPv4's depletion (we effectively cannot get more IPv4 space from any of the RIRs), b... [12:44:34] https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_get_security_level.html --> pretty interesting addition to OpenSSL 1.1 [12:50:32] elukey: do we have our own mirror for librdkafka packages? [12:51:18] we're currently using 0.11.3-1~bpo8+1+wikimedia1 [12:51:32] or is it just upstream (debian) compiled by us? [12:51:52] vgutierrez: Faidon maintains it in Debian: https://packages.qa.debian.org/libr/librdkafka.html [12:52:27] it's more of a team effort nowadays, but yeah, I help maintain it upstream [12:53:04] it looks like it needs a few patches regarding TLS configuration [12:53:11] I don't remember the deal around 0.11.3-1~bpo8+1+wikimedia1 but it's likely a backport to jessie compiled by us [12:53:32] in stretch we use 0.11.4-1~bpo9+1, which is a backport to stretch, upload to Debian's stretch-backports [12:53:43] uploaded* [12:56:05] vgutierrez: what kind of patches? [12:57:15] for instance.. right now you cannot choose the signature algorithms used.. so on the Client Hello it's sending some that are considered insecure like SHA1-DSA [12:57:33] are these changes upstream and need to be backported? [12:57:38] or are they not upstream at all? [12:57:57] not at all [12:58:03] ah [12:58:14] upstream is very responsive [12:58:31] I'd suggest to file a bug, with or without a patch :) [12:58:36] +1 [12:58:55] and if you don't get a reply in a few days ping me, and I'll reach out to him [12:59:14] awesome, thx <3 [13:00:37] sorry, I guess it might not have been obvious, but upstream is at https://github.com/edenhill/librdkafka/issues [13:01:09] yep yep.. I've been checking the code there :) [13:02:52] 10Traffic, 10Operations: Identify bots using AES128-SHA maintainers running on toolforge - https://phabricator.wikimedia.org/T194380#4210052 (10MaxBioHazard) usage: jsub [options...] program [args...] jsub: error: argument program: Program 'MONO_TLS_PROVIDER=btls' not found. [13:11:47] 10Traffic, 10Operations: Identify bots using AES128-SHA maintainers running on toolforge - https://phabricator.wikimedia.org/T194380#4210055 (10Vgutierrez) >>! In T194380#4210052, @MaxBioHazard wrote: > usage: jsub [options...] program [args...] > jsub: error: argument program: Program 'MONO_TLS_PROVIDER=btls'... [13:19:47] what is magnus doing these days? [14:11:59] works for confluence [14:12:03] librdkafka full time :) [14:12:43] confluence != Atlassian, right? [14:13:02] or maybe it's https://www.confluent.io/? [14:13:37] sorry, yes [14:14:11] he worked for the foundation in the past? [14:24:13] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4210178 (10fgiunchedi) WRT ms-fe servers (1008 and 1007), please move to asw2 and reallocate to be in two different physical racks. Ditto for ms-be machines,... [16:02:27] examples/rdkafka_example -P -b localhost:4433 -t test -d SECURITY -X security.protocol=SSL -X ssl.cipher.suites=ECDHE-ECDSA-AES256-GCM-SHA384 -X ssl.ca.location=ca/ca.pem -X ssl.certificate.location=ca/prototype-client.pem -X ssl.key.location=ca/prototype-client-key.pem -X ssl.curves.list=P-256 -X ssl.sigalgs.list=ECDSA+SHA256 [16:02:48] Signature Algorithms: ECDSA+SHA256 [16:02:53] yey.. it's working \o/ [16:06:09] now I should wrap the patch as a PR for librdkafka O:) [16:09:13] vgutierrez: awesome work :) [16:11:19] indeed, thanks! [22:42:05] ema, bblack, vgutierrez, fyi, had a switch rebooting in eqiad causing cp servers to loose connectivity. Chris replaced the power supplies and it now looks stable [22:42:12] https://phabricator.wikimedia.org/T194858 for the details