[01:40:31] 10Traffic, 10MobileFrontend, 10Operations, 10TechCom-RFC, 10Readers-Web-Backlog (Tracking): Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998 (10Krinkle) >>! In T214998#5029596, @dbarratt wrote: > Do we know how much of a burden it wi... [08:17:57] Krenair: I've marked your CR with the designate script as WIP in gerrit itself. So mark it as ready for review whenever it is ready :) [08:28:23] 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10jcrespo) labsdb1009.mgmt (stress on management interface) is down according to icinga for 14 hours (around net maintenance), maybe a loose cable o... [09:45:30] !log updated acme-chief to version 0.14 in acmechief[12]001 [09:45:30] vgutierrez: Failed to log message to wiki. Somebody should check the error logs. [11:23:45] 10netops, 10Operations, 10Patch-For-Review: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 - https://phabricator.wikimedia.org/T201039 (10jcrespo) Just to give an idea followup of es1014, issue seem gone: ` jynus@prometheus1004:~$ ping es1014.eqiad.wmnet PING es1014.eqiad.wmnet (10.64.16.187) 56(84) byt... [13:33:49] vgutierrez, oh right, forgot about that feature :) [13:33:51] ty [13:33:53] vgutierrez, wanted to talk to you about one of the TODOs in there [13:33:55] expiry of challenges [13:33:57] gdnsd handles this for us [13:33:59] but most DNS servers would not [14:09:46] right.. at the moment is not a priority for us. But as part of integrating OCSP stapling on the acme-chief side and getting rid of the hourly configuration reload I'd been thinking about implementing a hooks system based on the cert status.. so that could be an action to be triggered upon getting the valid certificates [14:24:47] it's also tricky from a statefulness perspective if the daemon restarts/reconfigues/crashes/etc [14:25:03] you could miss a cleanup trigger and leave some record stuck out there forever [14:26:00] indeed... regarding the specific issue raised by Krenair, I think that something external to acme-chief, cleaning old TXT records would be enough [15:16:25] true [15:17:05] I think designate does provide creation timestamps for recordsets actually so wouldn't even need to keep track of them, just a simple cron to clean up old ones should be sufficient [15:17:37] 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi) Thanks, opened T218789 [15:18:14] 10Traffic, 10MobileFrontend, 10Operations, 10TechCom-RFC, 10Readers-Web-Backlog (Tracking): Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998 (10Rillke) [16:30:09] Krenair: please check https://gerrit.wikimedia.org/r/c/operations/puppet/+/496148 PS7 when possible :) [18:40:40] Think I've found something odd with acme-chief [18:41:23] I'm not sure the HTTP_PROXY service env override is getting applied by puppet [18:42:20] uh? [18:42:41] well.. it's working cause otherwise in production we couldn't reach the LE endpoints [18:43:25] maybe something is funny about my setup [18:45:16] ah I think something in systemd::service didn't work the way I was expecting [18:46:34] ah I think something in systemd::service didn't work the way I was expecting [18:46:35] oops [18:55:20] https://phabricator.wikimedia.org/P8247 \o/ [18:55:47] acme-chief gets rather upset about the slow labs-ns0 - labs-ns1 update process but it gets there in the end [18:58:45] um, it might have broken designate though :| [18:58:46] wtf [21:16:02] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090 (10ayounsi) Typo above, test IP is 208.80.15**3**.225. Successfully tested on 1 link with: `cr4-ulsfo> ping source 129.250.204.6 208.80.153.225 ` Pushing the change to the ot... [21:29:47] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090 (10ayounsi) `name=cr2-codfw,lang=diff [edit interfaces xe-5/0/0] - description "Core: cr2-eqdfw:xe-0/1/4 (CyrusOne wikimedia:ix2.dfw4_to_ix2.dfw5.245.0009) {#11403} [10Gbps... [21:51:42] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090 (10ayounsi) Next step is to apply the following to replace the test IP with codfw text-lb IP. `lang=diff [edit firewall family inet filter transport-in4 term no-offload-ping4... [22:57:57] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Krenair) works now with some puppet cherry-picks