[00:05:32] 10netops, 10Operations: review and fix scs config - https://phabricator.wikimedia.org/T185926 (10ayounsi) 05Open>03Resolved a:03ayounsi Hey, I've done that long ago. All are also in LibreNMS and Rancid. I re-audited their config and only ulsfo had telnet enabled. We're all good here. [00:06:25] 10netops, 10Operations, 10decommission, 10ops-eqiad: unrack/decom pfw1-eqiad and pfw2-eqiad - https://phabricator.wikimedia.org/T183390 (10ayounsi) a:03Cmjohnson [00:47:58] 10netops, 10Operations: Security audit for tftp on install1001 - https://phabricator.wikimedia.org/T122210 (10ayounsi) 05Open>03Resolved a:03ayounsi To answer the original request, tftp isn't reachable from cloud, neither the internal subnet nor the public IPs are in the ACL. In addition Daniel confirmed... [01:06:42] 10netops, 10Cloud-VPS, 10Operations: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10ayounsi) Seems like even after T167357 I can reproduce the tests from the description. My guess is that it's setup like that to m... [09:29:10] 10netops, 10Operations: Intermittent connectivity issues in eqiad's row C - https://phabricator.wikimedia.org/T201139 (10Aklapper) [10:39:36] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp2013.codfw.wmnet', 'cp3030.esams.wmnet'] ``` The log can be found in `/var/l... [11:00:21] 10Traffic, 10Operations, 10Security-Team, 10Wikimedia-General-or-Unknown: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618 (10ema) p:05Triage>03Normal [11:00:31] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10ema) p:05Triage>03Normal [11:01:00] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-General-or-Unknown, and 2 others: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest - https://phabricator.wikimedia.org/T199252 (10ema) p:05Triage>03Normal [11:50:45] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2013.codfw.wmnet', 'cp3030.esams.wmnet'] ``` and were **ALL** successful. [12:01:59] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp2014.codfw.wmnet', 'cp3033.esams.wmnet'] ``` The log can be found in `/var/l... [12:03:03] 10netops, 10Cloud-VPS, 10Operations: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10aborrero) a:03aborrero Our plan is to keep using the `dmz_cidr` mechanism with the new `172.16` addressing space. This is alrea... [12:09:38] bblack: https://gerrit.wikimedia.org/g/operations/software/certcentral/+/refs/changes/45/454845/7/certcentral.py#83 [12:10:48] this CR implements DNS-01 and allows picking one validation method (http-01 or dns-01) as a config setting for each certificate [12:11:16] it also bring integration tests with dns-01 using pebble and a tiny tiny DNS server based on dnslib [12:33:45] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2014.codfw.wmnet', 'cp3033.esams.wmnet'] ``` and were **ALL** successful. [13:17:44] ok [13:18:15] looks like the param format is something like: example.org sha1stuff www.example.org sha1stuff .... [13:18:19] which should work great [13:18:49] to be sure we're on the same page about sha1stuff (or in other words, to make sure I didn't misread the spec!) [13:19:14] (actually it's not even sha1, now that I go look at my notes, of course) [13:20:35] I'm expecting it to be a SHA-256 output (256-bits or 32 bytes, in binary form), encoded as "base64url" encoding [13:22:12] ... and base64url gives us 6 bits per byte, so should be... ~43 bytes of output, with I guess some zero-pad there at the end [13:23:05] (43 bytes of base64url encodes 258 bits of data, so there's 2 extra bits there at the end) [13:25:04] hmm maybe I need to go re-read base64url stuff, it seems like existing generators make 44 bytes of it [13:28:22] right, I didn't understand the padding rules [13:32:52] yeah, so standard base64url says padding is optional if the length is known (it is in this case, I think) [13:32:59] so it might just be 43 bytes [13:33:53] ah, and the acme spec says to strip trailing =-padding [13:33:57] 43 bytes it is! [13:35:21] and indeed, their examples have 43 bytes as well [13:37:16] ok [13:37:55] example.org <43 bytes of b64url chars> www.example.org <43 bytes of b64url chars> .... [13:42:24] trying to stay under ARG_MAX :-P (it's so large nowadays to not be an issue ofc) [15:36:33] 10netops, 10Cloud-VPS, 10Operations: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10ayounsi) So first, why maintain 4 different lists instead of 1? (or at least have the same subnets in each lists). Then 185.15.56.... [15:45:43] vgutierrez, hey [15:45:55] just looking at the bottom of the list of commits [15:46:07] "Refactor certcentral.certificate_management()" is looking good [15:46:23] I do notice that since PS11 you've changed some enum equality checks with identity checks [15:46:51] which we can probably get away with but I'm not sure it's a good idea in general [17:21:46] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Redirect 2030.wikimedia.org to the new movement strategy portal - https://phabricator.wikimedia.org/T202498 (10Dzahn) The changes has been deployed. The redirect should change once varnish cashes are updated. ( i think within 24 hours max) [17:47:38] 10netops, 10Cloud-VPS, 10Operations: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10chasemp) Yeah it would be best to have this list of prod networks to preserve 172 source IP for: a) a fixed list of required end... [17:52:38] 10netops, 10Cloud-VPS, 10Operations: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10Krenair) [22:43:53] https://labs.ripe.net/Members/willem_toorop/sunrise-dns-over-tls-sunset-dnssec