[00:25:29] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-Incident: 15% response start regression as of 2019-11-11 (Varnish->ATS) - https://phabricator.wikimedia.org/T238494 (10Krinkle) [07:03:40] 10netops, 10Operations, 10Patch-For-Review: Re-prioritize peering over transit - https://phabricator.wikimedia.org/T259614 (10ayounsi) 05Open→03Resolved All done! [11:36:28] 10Traffic, 10Operations, 10SRE-tools, 10Goal, and 3 others: Automate generation of Management DNS records from Netbox - https://phabricator.wikimedia.org/T233183 (10Volans) 05Open→03Resolved a:05crusnov→03Volans All management records are now generated via Netbox, related wikitech documentation upd... [13:06:39] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Vgutierrez) hmm that's interesting, please note that this is not the first time we use websockets. etherpad.wm.o i... [13:22:25] vgutierrez: https://gerrit.wikimedia.org/r/619465 [13:23:01] cdanis: yeah.. that's exactly what I was going to submit now [13:23:09] facepalmed when I realized [13:48:58] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10CDanis) The Envoy TLS terminator is now configured to allow websocket upgrades -- however, it's improperly configu... [16:22:57] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [16:38:31] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10Cmjohnson) @ayounsi the cross-connect has been completed but I am not seeing any light from the demarc panel [16:48:23] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10Cmjohnson) just to confirm, I double-checked the Equinix completion email and verified the ports (11,12) and the patch number (21504182-A). I also verified the sfp+ and fiber ar... [17:46:04] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10Cmjohnson) [17:58:29] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10RobH) We've not gotten confirmation from Cloudflare that they are turned up, I'll email them to let them know we are ready on our end! [18:33:32] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10RobH) a:05Cmjohnson→03RobH I just emailed back and forth with Matt @ Cloudflare (he is very prompt in replies!). They had to put in an EQ order to have a patch placed between... [18:35:01] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10RobH) 05Open→03Resolved >>! In T259923#6377508, @RobH wrote: > I just emailed back and forth with Matt @ Cloudflare (he is very prompt in replies!). > > They had to put in an... [20:10:39] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) test comment [20:16:17] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) Will it blend? [20:19:30] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) it appears to blend. [20:36:22] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) >>! In T238593#6376059, @CDanis wrote: > The Envoy TLS terminator is now configured to allow websocket upgr... [20:44:07] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) 05Open→03Resolved a:03Dzahn We are seeing realtime notifications again and aphlict is now separated f...