[06:16:26] <wikibugs>	 10DBA, 10Operations: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10Marostegui) Don't worry, as soon as we arrange a date/time, I will stop it, so we are sure that no lag will happen before the failover. I will leave the screen running and just kill the process so you can...
[06:28:30] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Upgrade firmware on db1078 - https://phabricator.wikimedia.org/T209815 (10Marostegui) Chris, can we do this today as this host will be the future s3 primary master?
[07:44:26] <wikibugs>	 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui)
[08:25:02] <wikibugs>	 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui)
[08:50:28] <marostegui>	 hey jynus we've got a bunch of things ahead :(
[08:50:42] <marostegui>	 Can you take a look at: https://gerrit.wikimedia.org/r/#/c/operations/dns/+/484611/
[08:51:59] <marostegui>	 thanks!
[08:52:35] <marostegui>	 next question….tomorrow 8AM (7AM UTC) for s3 failover would work for you?
[08:52:40] <marostegui>	 I have made all the patches for db1078 already
[08:52:48] <marostegui>	 https://phabricator.wikimedia.org/T213858
[08:54:42] <jynus>	 are managers ok with that?
[08:55:05] <marostegui>	 they were yesterday on the hangouts at around 10pm
[08:55:11] <marostegui>	 let me ask them
[08:55:40] <jynus>	 I asked them but got no definitive answer
[08:56:33] <marostegui>	 just asked
[09:01:50] <jynus>	 for some reason I got disconnected
[09:03:40] <jynus>	 should I restart and upgrade the replicas?
[09:03:48] <marostegui>	 I finished codfw just now
[09:03:54] <marostegui>	 if you want to take the eqiad ones
[09:03:57] <marostegui>	 I think there are not many left
[09:04:03] <marostegui>	 I did some during christmas
[09:04:14] <marostegui>	 And later re-check our etherpad and the patches
[09:04:18] <marostegui>	 sorry for all the homework :(
[09:05:08] <jynus>	 I think I have connection issues
[09:08:38] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10Marostegui) a:03Marostegui This has been done
[09:10:05] <jynus>	 check ongoing connections throught the proxy, some services may need a restart
[09:10:20] <marostegui>	 yeah, I was wondering if phab needs so
[09:10:54] <jynus>	 let them die naturally first, but if it still has tcp connections, it has to be restarted
[09:11:01] <marostegui>	 yep
[09:11:35] <marostegui>	 there is nothing from what I can see
[09:12:56] <marostegui>	 on mysql itself I see just 2 connections
[09:13:00] <marostegui>	 coming from dbproxy1003
[09:13:05] <marostegui>	 let's give it some time
[09:13:08] <marostegui>	 they are sleeping 
[09:14:11] <jynus>	 backups may have had issues because tendril restart
[09:14:27] <jynus>	 this is connected due to restart of backup sources
[09:14:44] <marostegui>	 yeah
[09:16:04] <jynus>	 I will later test the switchover script on test-s4
[09:16:20] <marostegui>	 thanks
[09:20:46] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) >>! In T212487#4867178, @Marostegui wrote: > @elukey For those databases that...
[09:31:14] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10Marostegui) @Anomie I have stopped the script as we are most likely going to go ahead with the failover in EU morning (still waiting for the managers to confirm)
[10:35:21] <jynus>	 I think I need more coffee to run mysql_upgrade on s3
[10:35:43] <marostegui>	 haha
[10:36:13] <jynus>	 db1077 is partially repooled, will go with db1095 next while I wait for it to heat up
[10:36:33] <jynus>	 and later db1123
[10:36:47] <marostegui>	 ok
[10:37:16] <marostegui>	 I will wait for cmjohnson1 for https://phabricator.wikimedia.org/T209815 to be done later when he is onsite
[10:37:58] <jynus>	 I was wondering if to reboot it
[10:38:22] <marostegui>	 let's wait for the firmware upgrade
[10:38:27] <marostegui>	 so we can do it all at once
[10:39:26] <jynus>	 I will try to have all other hosts in a good state by then
[10:39:53] <jynus>	 I didn't start es1019
[10:40:01] <jynus>	 not sure if we should
[10:40:36] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10Marostegui) @mmodell ` ˜/marostegui 11:39> twentyafterfour: We have failed over dbproxy1003 to dbproxy1008 which is phabricator, I still see two connections (they have been there for ho...
[10:40:39] <marostegui>	 I think chris is done with it
[10:41:09] <jynus>	 I will renew the downtime at least until friday
[10:41:16] <marostegui>	 oki
[10:52:09] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10Marostegui) >>! In T213865#4883806, @Marostegui wrote: > @mmodell > ` > ˜/marostegui 11:39> twentyafterfour: We have failed over dbproxy1003 to dbproxy1008 which is phabricator, I still...
[11:20:21] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[11:24:16] <jynus>	 db1068 correctable errors seems to have gone down, so I guess that is good?
[11:24:33] <marostegui>	 yeah, same behaviour happened in decemeber
[11:24:34] <marostegui>	 december
[11:24:39] <marostegui>	 and one day they were fully gone
[11:25:05] <jynus>	 I created the task to keep it in mind, will close it if they go to 0
[11:28:42] <cmjohnson1>	 I am done with es1019.  I forgot to close the task.   I will update it in a few minutes.
[11:29:10] <jynus>	 np, cmjohnson1 we thought you were busy enough
[11:29:25] <jynus>	 but didn't want to do sth without your green light
[11:40:13] <jynus>	 going for db1095 next
[12:01:58] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10jcrespo) So this is solved?
[12:25:03] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10Marostegui) Yes
[12:25:26] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Failover dbproxy1003 to dbproxy1008 - https://phabricator.wikimedia.org/T213865 (10Marostegui) 05Open→03Resolved
[12:52:01] <marostegui>	 jynus: let me know when I can depool db1078 
[12:53:26] <jynus>	 if you are going to deploy to eqiad-php
[12:53:35] <jynus>	 we need to merge the repool first
[12:53:50] <jynus>	 I was waiting for swat to finish
[12:53:57] <marostegui>	 sure, no worries
[12:54:03] <marostegui>	 I need to coordinate with cmjohnson1 still
[12:54:27] <jynus>	 so should I depool db1123 or do I wait?
[12:54:35] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10Marostegui) Just to confirm:  Date: Thursday 17th January Time: 07:00 AM UTC - 07:30 AM UTC (we expect not to use the full 30 minutes window)  **Impact: All those wikis will go read-o...
[12:54:47] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10Marostegui)
[12:55:17] <marostegui>	 I think you can go with db1123 as I haven't arranged a time with Chris yet
[12:55:33] <cmjohnson1>	 Marostegui: what do we need to coordinate?
[12:56:36] <marostegui>	 cmjohnson1: db1078 will be the new db master for s3 tomorrow, so I would like to get its firmware upgraded today if you have time: https://phabricator.wikimedia.org/T209815
[12:57:06] <cmjohnson1>	 Sure. Take it offline and I will do now
[12:57:26] <marostegui>	 ok, let me coordinate with jynus XD
[12:57:29] <marostegui>	 jynus: ^
[13:00:17] <marostegui>	 cmjohnson1: we are going to depool db1078 and I will ping you once it is off
[13:00:32] <cmjohnson1>	 Ok
[13:17:12] <marostegui>	 cmjohnson1: db1078 is now off, you can proceed
[13:21:33] <jynus>	 I will give es1019 a kick
[13:22:01] <cmjohnson1>	 Great!
[13:22:21] <marostegui>	 cmjohnson1: when done turn db1078 off and I will take it from there :)
[13:22:25] <marostegui>	 *on
[13:22:35] <jynus>	 marostegui: did you do a full-upgrade?
[13:22:37] <marostegui>	 yep
[13:24:03] <jynus>	 we will see if mgmt lasts this time...
[13:32:09] <cmjohnson1>	 marostegui and jynus...sorry this is going to take a little longer than I thought. I forgot it was a HP server. I have to run the service pack but I need to be there to do it.....I'm on my way now but will be about 1 hour w/traffic 
[13:32:44] <marostegui>	 cmjohnson1: no worries
[13:32:51] <marostegui>	 I will extend the downtimeç
[13:33:16] <jynus>	 I can do db1124
[13:33:38] <marostegui>	 cool
[13:33:51] <jynus>	 it will affect several hosts but we have to do at some poinmt
[13:34:02] <marostegui>	 yeah
[13:34:09] <jynus>	 and better now without s3 updates
[14:08:42] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10elukey) @Marostegui I have now everything in my home gzipped, I'll move it to stat1007 and...
[14:09:27] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) cool, let me know when you think it is safe for me to delete my files
[14:09:52] <elukey>	 you can delete anytime --^
[14:09:57] <marostegui>	 ok thanks
[14:13:31] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) Just for the record, I am spending an incredible amount of time fixing dbstore1002 on the last few days si...
[14:13:41] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[14:14:27] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) As spoken on IRC...I have deleted my files.
[15:10:44] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1115 (tendril DB) had OOM for some processes and some hw (memory) issues - https://phabricator.wikimedia.org/T196726 (10jcrespo) 05Open→03Stalled stalling, no errors so far, but I doubt this is the last time we hear abut this. Backups are on dbstore1001 just in case.
[15:12:03] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Upgrade firmware on db1078 - https://phabricator.wikimedia.org/T209815 (10Cmjohnson) I ran the Service Pack on db1078, all firmware is up to date including BIOS and raid controller.  The server is currently powered off
[15:12:56] <marostegui>	 cmjohnson1: will you power it on for us?
[15:12:59] <marostegui>	 or do you want me to?
[15:13:10] <cmjohnson1>	 i can power it on if you are ready for it
[15:13:16] <cmjohnson1>	 you asked to leave it powered off
[15:13:20] <marostegui>	 yep! go for it
[15:13:23] <cmjohnson1>	 ok
[15:13:26] <marostegui>	 yeah, it was a typo, i wanted to say "on" XD
[15:13:28] <marostegui>	 Sorry!
[15:13:52] <cmjohnson1>	 hah..no worries it's coming up now
[15:13:55] <marostegui>	 thank you!
[15:16:29] <cmjohnson1>	 marostegui are you able to ssh?
[15:17:11] <marostegui>	 yes! it works
[15:17:17] <marostegui>	 I will take it from here
[15:17:20] <marostegui>	 Thanks a lot
[15:17:20] <cmjohnson1>	 cool
[15:17:47] <marostegui>	 I will resolve the task once I repool it
[15:37:37] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Upgrade firmware on db1078 - https://phabricator.wikimedia.org/T209815 (10Marostegui) 05Open→03Resolved Thank you so much! The server is back in the mix.
[17:29:21] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10jcrespo) switchover script works as expected (tested on db1111/db1112):  `lang=sh, lines=10 ./switchover.py --skip-slave-move db1111 db1112 Starting preflight checks... * Original rea...
[17:35:59] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: s3 master emergency failover (db1075) - https://phabricator.wikimedia.org/T213858 (10Marostegui) Awesome news! We have to include it on the steps list on our etherpad, which I wrote yesterday evening and needs to be reviewed by you, as it was late in the day, so error...
[17:38:26] <jynus>	 ^I just did that
[17:38:49] <jynus>	 put the things not needed on "Equivalent manual steps" to skip
[17:41:01] <jynus>	 I will also try for next time to do the topology changes automatically integrating the repl functionality
[17:43:51] <marostegui>	 thank you!
[17:43:55] <marostegui>	 i have a question
[17:44:00] <marostegui>	 about your line 33
[17:44:06] <marostegui>	 check my line 34 :)
[17:47:08] <jynus>	 I want to asume that either we abort or go through
[17:47:34] <jynus>	 so we leave gtid and semisync enabled before the actual failover
[17:47:53] <jynus>	 well, I don't know if I want it
[17:47:56] <jynus>	 I am proposing it
[17:48:20] <marostegui>	 But we always disable gtid
[17:48:24] <marostegui>	 You want to try not doing so?
[17:48:35] <jynus>	 yes, we disable it for migration
[17:49:18] <jynus>	 it is a small thing so no big deal
[17:49:25] <jynus>	 as long as we leave it for later
[17:49:35] <marostegui>	 yeah, it is on the list, to put it back ON
[17:49:43] <jynus>	 what I don't want is to keep users waiting for it
[17:49:49] <jynus>	 so either reenable it in advance
[17:49:57] <jynus>	 or doit as the last thing
[17:49:57] <marostegui>	 users waiting for what?
[17:50:31] <jynus>	 outside of the "critical time"
[17:50:56] <marostegui>	 I am not sure what you mean, why would users wait for GTID?
[17:51:08] <jynus>	 don't worry, we can talk tomorrow
[17:51:49] <marostegui>	 sure
[21:24:44] <wikibugs>	 10DBA, 10Fundraising-Backlog: Remove frimpressions db from prod mysql - https://phabricator.wikimedia.org/T213973 (10cwdent)
[21:49:57] <wikibugs>	 10DBA, 10Fundraising-Backlog: Remove frimpressions db from prod mysql - https://phabricator.wikimedia.org/T213973 (10awight) How many rows are in the tables?  There should be a timestamp column, often `ts`, to query.  It's probably worth keeping an archival dump if it might be real data.
[23:30:06] <wikibugs>	 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Pchelolo) 05Resolved→03Open Hm, trying to deploy the service again I see the same issue:  `...
[23:40:12] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10greg) >>! In T212487#4867200, @Marostegui wrote: > I asked @chasemp about `fab_migration`...