[12:31:43] <milimetric>	 apologies to halfak and Nettrom for missing pings, I'll be off for an hour or so and then back on.  Might want to start a phab task or an email as this seems like a longer design brainstorm than an IRC chat can do justice
[17:29:18] <J-Mo1>	 halfak are you coming to documentation time today? I've got a couple things I wanted to run by you
[17:29:34] <halfak>	 J-Mo1, I'm in CSCW paper mode, but I think I can be there anyway. 
[17:29:54] <J-Mo1>	 cool. I'll keep it brief
[19:56:28] <mutante>	 halfak: Amir1: hi, are you online to watch ores a little
[19:56:45] <mutante>	 i'll change it back to how it was before.. right
[19:56:53] <mutante>	 but i was told i should not have done this in one hard step
[19:57:11] <mutante>	 but in a soft-switch, first active/active, wait, then disable the original one
[19:57:31] <mutante>	 so i'll do that when going back
[19:59:13] <halfak>	 mutante, did something go wrong?
[20:00:49] <mutante>	 halfak: no. i mean something went wrong yesterday after deploy, so we switched to eqiad, right. and now codfw is fixed by amir and so we would revert to how it was before 
[20:01:13] <halfak>	 Oh I see.  Yes I'll be around for the next few hours. 
[20:01:18] <halfak>	 To monitor and help out :) 
[20:02:23] <mutante>	 halfak: ok, great. so what i will do now is set it to BOTH for like half an hour
[20:02:33] <mutante>	 and then disable codfw
[20:02:47] <mutante>	 then everything is like before, and this is what traffic told me, do it in 2 steps
[20:02:55] <mutante>	 like, i should have done that yesterday too
[20:02:59] <halfak>	 mutante, s/codfw/eqiad/
[20:03:04] <mutante>	 to avoid 50x during switch
[20:03:13] <mutante>	 arg, yes :) 
[20:03:30] <mutante>	 back to how it was before the deployment yesterday
[20:03:41] <mutante>	 and then i will leave it like that until the Ops "switch DC" period is over
[20:04:02] <mutante>	 and then we finally set it to active/active and leave it like that permanently
[20:05:37] <halfak>	 OK cool.  Sounds like a good plan. 
[20:05:54] <halfak>	 mutante, why would that make sense to do yesterday? 
[20:05:58] <halfak>	 When codfw was broken. 
[20:07:32] <mutante>	 halfak: good question, frankly i was just adviced to do it like that by a comment from brandon of traffic
[20:09:03] <mutante>	 15:59 < bblack> mutante: that method of switching causes a 5xx spike (but luckily this is a low-traffic service)
[20:09:06] <mutante>	 15:59 < bblack> the duration of the 5xx spike should only be during the cumin run, basically
[20:09:09] <mutante>	 16:02 < bblack> (sorry it's not well documented outside of wikitech.  But basically to switch from only-eqiad to only-codfw (or vice-versa) without a 508 spike during the deployment process, you 
[20:09:13] <mutante>	                 have to do it in two stages, switching to active:active first in the middle)
[20:09:16] <mutante>	 well, and only during the cumin run .. which isnt that long
[20:09:51] <mutante>	 508 is "loop detected"
[20:11:13] <halfak>	 gotcha.  That makes sense. 
[20:11:35] <halfak>	 In future situations like that, we might want to go active-active only for a brief moment. 
[20:14:29] <mutante>	 doing the puppet run on the 15 cache servers , that makes it active/active
[20:17:38] <mutante>	 finished
[20:18:00] <halfak>	 cool.  Checking on grafana
[20:18:37] <halfak>	 Saw codfw come alive at 2008 UTC. 
[20:18:52] <halfak>	 Errors and response time looks OK
[20:19:06] <mutante>	 sounds good
[20:23:17] <halfak>	 Everything continues to look good. Looking away for a bit. 
[20:23:58] <mutante>	 ok, then uploading the second step / rebasing Amir's change to revert my original change
[20:32:30] <mutante>	 switching back to codfw-only 5/15
[20:33:19] <halfak>	 5/15/
[20:33:21] <halfak>	 *?
[20:34:50] <mutante>	 done. 15/15 (servers that are running varnish and make up the "misc:cache" group)
[20:34:59] <halfak>	 Oh I see
[20:35:15] <mutante>	 yea, when i run puppet there to apply the config change on all
[20:35:32] <mutante>	 so now we are back to like it was before yesterday's deploy
[20:45:47] <halfak>	 mutante, all looks good. 
[20:46:51] <halfak>	 Ready to declare victory?
[21:21:23] <lzia>	 hey halfak. a note that I've received your ping about Tech Dept. Platforms. I just had not a chance to give it a close look. Once I do that, I will address your assignment to me as well. Thanks for the ping.
[21:24:15] <halfak>	 lzia, hey!  No problem.  Just wanted to make sure your stuff got listed in whatever kind of thing that document will become :) 
[21:27:39] <lzia>	 yeah, really appreciate it, halfak. :) it's been non-stop since a few days ago. ;)
[21:28:33] <halfak>	 I hear you.  Keep on trucking.  It's been the same here.  Just getting done with paper #2.  Had a major hiccup in data processing today that threatened the submission.  D:  
[21:32:10] <lzia>	 :D
[21:32:14] <lzia>	 good luck, halfak. :)
[21:32:24] <halfak>	 Thanks 
[21:32:30] * halfak flexes and digs in heels. 
[21:32:39] <halfak>	 you too!
[21:42:51] <lzia>	 yeah, thanks.
[22:29:29] <mutante>	 halfak: yes, victory , all good :)
[22:29:47] <halfak>	 :)