[01:59:30] <wikibugs>	 10netops, 10Operations: ulsfo <-> router1.corp BGP sessions down - https://phabricator.wikimedia.org/T217207 (10eliza) hard boot router1 and now it's up.  Find attached the SysLog  {F28292104}
[02:03:42] <wikibugs>	 10netops, 10Operations: ulsfo <-> router1.corp BGP sessions down - https://phabricator.wikimedia.org/T217207 (10ayounsi) 05Open→03Resolved
[02:26:34] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10VisualEditor: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10matmarex) May be related to using IPv6? https://www.mediawiki...
[03:58:22] <wikibugs>	 10Traffic, 10ExternalGuidance, 10Operations, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Deliver mobile-based version for automatic translations - https://phabricator.wikimedia.org/T212197 (10santhosh) >  @santhosh is there any scenario in a future architecture of the extension e whe...
[08:45:08] <wikibugs>	 10Acme-chief, 10Traffic, 10Operations: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez)
[08:45:21] <wikibugs>	 10Acme-chief, 10Traffic, 10Operations: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez) p:05Triage→03High
[08:53:00] <wikibugs>	 10Acme-chief, 10Traffic, 10Operations, 10Patch-For-Review: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez) 05Open→03Resolved Thanks @Volans for reporting the issue <3
[08:53:58] <volans>	 vgutierrez: <8
[09:02:07] <wikibugs>	 10Acme-chief, 10Traffic, 10Operations: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez)
[10:21:39] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10Gilles) Did something happen on 2019-02-21? I was looking at the ulsfo RUM perf metrics for the change in that DC. No change is apparent after the 2019-02-19 config change...
[11:54:20] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10BBlack) Circa 2019-02-21, eqsin was depooled to install a new router, and most of the users normally mapped to eqsin had fallen back to ulsfo temporarily, which would dist...
[13:05:36] <onimisionipe>	 vgutierrez: https://gerrit.wikimedia.org/r/c/operations/puppet/+/493048 one last pass :) hopefully
[13:05:42] <onimisionipe>	 for when you have some time
[13:21:16] <vgutierrez>	 after lunch please :)
[13:21:38] <vgutierrez>	 I'm trying to survive my gym daily session
[13:55:20] <onimisionipe>	 vgutierrez: No p!
[13:55:48] * onimisionipe wonders why vgutierrez needs gyming
[14:37:11] <vgutierrez>	 onimisionipe: looking good from my side
[14:37:16] <vgutierrez>	 (the CR)
[15:19:47] <onimisionipe>	 vgutierrez: Thanks!
[15:31:21] <wikibugs>	 10HTTPS, 10Traffic, 10Operations: Make sure that services available for NDA-only users are using strong TLS ciphersuites - https://phabricator.wikimedia.org/T217002 (10Vgutierrez)
[16:24:07] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson) We have an issue with these servers. I completely forgot about the lack of power availability in rack A2.   We will need to move these to a different row or wait...
[16:25:38] <onimisionipe>	 vgutierrez: I got this when I ran puppet on relforge
[16:25:43] <onimisionipe>	 https://www.irccloud.com/pastebin/8kdCvzVo/
[16:25:59] <onimisionipe>	 acme setup times out
[16:26:25] <onimisionipe>	 not sure why
[16:26:26] <vgutierrez>	 well... you cannot get an LE certificate for an eqiad.wmnet hostname :)
[16:26:40] <onimisionipe>	 ouch!
[16:26:41] <vgutierrez>	 .wmnet is an internal DNS zone not valid on the internet
[16:26:46] <onimisionipe>	 it has to be public
[16:26:47] <vgutierrez>	 we discussed this before :)
[16:26:49] <vgutierrez>	 indeed
[16:27:35] <onimisionipe>	 alright. Thanks!
[16:28:20] * onimisionipe now gets the LE thingy
[17:31:47] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson)
[17:32:11] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson)
[17:33:36] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) >>! In T212348#4988345, @Cmjohnson wrote: > We have an issue with these servers. I completely forgot about the lack of power availability in rack A2.   We will need t...
[17:48:57] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) Setup new ports for the mw systems in a5:    ` [edit interfaces interface-range disabled] -    member ge-5/0/1; -    member ge-5/0/0; -    member ge-5/0/2; -    membe...
[17:55:46] <wikibugs>	 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) ports in a7 for ms-be and ms-fe setup, using ports that were for old mw systems on old switch stack and will be unused on new stack:  robh@asw2-a-eqiad# show | compar...
[19:13:46] <wikibugs>	 10Traffic, 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) This is still caus...
[19:23:19] <wikibugs>	 10Traffic, 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) Added a new deploy...
[21:02:33] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) cr1/2-codfw + cr2-eqdfw `lang=diff [edit policy-options policy-statement BGP_community_actions] -    term peer-private-peer { -        from community PEER_PRIVATE...
[21:24:19] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Because of the way codfw/eqdfw peers with eqiad and eqord and the fact we have similar peers in several IXPs Removing the preferred local-pref in Dallas caused th...
[22:10:19] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Same config diff for eqord/eqiad  eqord: Transit: https://librenms.wikimedia.org/graphs/id=16837,16841/type=multiport_bits_separate Peering: https://librenms.wiki...
[22:46:41] <wikibugs>	 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Shift from peering to transit are: ~200Mbps in eqdfw ~300Mbps in eqord ~300Mbps in eqiad
[23:14:34] <mobrovac>	 bblack: quick q: when a req for mediawiki comes to varnish, it always routes it to eqiad unconditionally? can you point me to the vcl code handling that?
[23:15:10] <mobrovac>	 for context, i'm trying to figure out the changes needed to vcl if we want to route get/head requests to the closest dc rather than the master dc
[23:29:36] <volans>	 mobrovac: to answer the first part, it's a bit more complex than that, as we have multiple layers of varnishes (within the same DC and from PoPs to core DCs)
[23:29:52] <mobrovac>	 yup yup, but i
[23:30:02] <volans>	 to simplify it a bit (as I'm sure I would anyway miss some details on the detailed picture)
[23:30:05] <mobrovac>	 i'm basically interested in the b-e varnishes 
[23:30:42] <volans>	 it all boils down to, when reaching a core DC, if the service is marked as active/active or active/passive to know if it can be routed locally or needs to go to the other core DC
[23:31:20] <mobrovac>	 ah right
[23:31:32] <mobrovac>	 huh ok, so the logic here will need to be complicated
[23:31:42] <volans>	 this is the gloabl config for the text cluster for example
[23:31:43] <volans>	 https://github.com/wikimedia/puppet/blob/production/hieradata/role/common/cache/text.yaml#L12
[23:31:50] <mobrovac>	 since we will want get/head to go to the local dc, and posts to go to the main dc
[23:32:22] <mobrovac>	 ah yes, i know that yaml very well :)
[23:32:37] <mobrovac>	 huh ok, we'll need to invest time here then
[23:32:44] <volans>	 now, how that can be translated into VLC to achive what you want... I have no idea :)
[23:33:14] <volans>	 and I'm sure bran.don / e.ma have already thought about this
[23:35:04] <volans>	 my guess, but I'm totally guessing, is thtat from the hiera point of view it will be active/active as the appropriate directors must be created in VLC to be able to route both local and remote DC and then some magic will be done inside the VCL to decide what to do
[23:35:23] <volans>	 based on the request's details
[23:36:24] <mobrovac>	 hm yeah, that does seem like a viable way
[23:36:31] <mobrovac>	 albeit, it's a work-around rather than a solution
[23:36:48] <mobrovac>	 go to sleep volans! :P
[23:36:52] <mobrovac>	 thnx for your help :)
[23:36:54] <mobrovac>	 really appreciate it
[23:37:32] <volans>	 unless I just said random things and the reality is completely different :)
[23:37:42] <volans>	 and the fun part is that's totally possible :-P
[23:37:49] <mobrovac>	 :)))))
[23:37:51] <volans>	 anyway... yes, time to go to be
[23:37:53] <volans>	 *bed
[23:38:09] * mobrovac tucks volans in
[23:38:11] <mobrovac>	 nighty night
[23:38:16] <volans>	 lol
[23:38:22] <mobrovac>	 :P