[07:16:26] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Marostegui)
[07:23:00] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Marostegui) p:05Triage→03High BBU broke `    Battery/Capacitor Count: 0 ` @Cmjohnson Can we give this host some priority? I wouldn't want to have it down for the whole offsite week. I believe its support just...
[07:24:25] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Marostegui) @jcrespo I am going to place db1135 temporarily (T222682) to replace this host until we have found a solution
[07:46:24] <wikibugs>	 10DBA, 10Goal: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[08:02:01] <jynus>	 thanks for the quick reaction
[08:02:35] <marostegui>	 jynus: I am going to leave mysql stopped for now, until we hear from cmjohnson1 , once that is done, I will start mysql and check the data
[08:02:40] <marostegui>	 sounds good?
[08:02:51] <marostegui>	 I doubt cmjohnson1 will have a spare BBU, but let's wait
[08:03:02] <marostegui>	 the support finished the 24th may, bad luck :(
[08:03:51] <jynus>	 I am not sure of that
[08:03:55] <jynus>	 let me check
[08:04:16] <marostegui>	 not sure of what?
[08:05:52] <jynus>	 indeed they arrived 3 years ago
[08:11:35] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Marostegui)
[08:36:55] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/514439/
[08:38:17] <wikibugs>	 10DBA, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo) @Marostegui Ok if I send them to DC Ops?
[08:38:34] <wikibugs>	 10DBA, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10Marostegui) +1
[08:52:50] <jynus>	 I am going to try a restore before full decom
[08:53:50] <marostegui>	 sure
[08:54:02] <marostegui>	 can you take a look at the above patch for db1135 when you have a minute?
[09:15:46] <wikibugs>	 10DBA, 10Goal: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[09:22:06] <jynus>	 how did you set it up?
[09:22:45] <marostegui>	 with a snapshot
[09:22:55] <jynus>	 cool
[09:23:01] <marostegui>	 :)
[09:23:11] <jynus>	 I am recovering to dbprov2001 and 2
[09:23:30] <jynus>	 FYI in case something goes wild like consuming 10 TB of data or something
[09:23:39] <marostegui>	 good test indeed
[09:23:40] <wikibugs>	 10DBA, 10Goal: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[09:23:51] <jynus>	 the state is a bit worring, will tell later
[09:24:25] <marostegui>	 sure
[09:25:52] <marostegui>	 it has been weeks since I was the DBQuery so clean XD
[09:26:03] <marostegui>	 the DBQuery logstash channel
[09:26:07] <jynus>	 indeed
[09:26:20] <marostegui>	 s/was/saw
[09:26:20] <jynus>	 except for the 400K when hw crashes
[09:26:27] <jynus>	 :-)
[09:26:32] <marostegui>	 haha yeah
[09:27:27] <jynus>	 actually, aside from the 17 ongoing queries
[09:27:30] <jynus>	 no fatals
[10:01:18] <wikibugs>	 10DBA, 10Operations, 10observability: Generate instance list of database hosts to be monitored automatically from exported resources - https://phabricator.wikimedia.org/T177779 (10fgiunchedi) I was reviewing #observability backlog, to me it looks like this is a duplicate of {T145072} ?
[10:03:35] <wikibugs>	 10DBA, 10Operations, 10observability: Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb - https://phabricator.wikimedia.org/T145072 (10jcrespo)
[10:03:38] <wikibugs>	 10DBA, 10Operations, 10observability: Generate instance list of database hosts to be monitored automatically from exported resources - https://phabricator.wikimedia.org/T177779 (10jcrespo)
[10:05:13] <wikibugs>	 10DBA, 10Operations, 10observability: Generate instance list of active database hosts to be monitored from prometheus - https://phabricator.wikimedia.org/T145072 (10jcrespo)
[10:18:09] <wikibugs>	 10DBA, 10MediaWiki-extensions-Renameuser: Fix use of DB schema so RenameUser is trivial - https://phabricator.wikimedia.org/T33863 (10Aklapper)
[10:29:57] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global renaming of B_dash → A1Cafel - https://phabricator.wikimedia.org/T224916 (10revi) @jcrespo please consider sending an email to global-renamers - at - lists.wikimedia.org in case of improvements like this -- which everyone will love.
[10:36:40] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global renaming of B_dash → A1Cafel - https://phabricator.wikimedia.org/T224916 (10jcrespo) @revi I will suggest the developers to announce it there (all credit to the core team), although I am mostly sure they are aware of this; just not ready enough to be announced yet.
[12:25:20] <jynus>	 restore works as intended, although there are many thing I don't like
[12:26:21] <marostegui>	 like?
[12:26:35] <jynus>	 retention and purging, and what that entails
[12:26:44] <jynus>	 also not a clear recovery path due to the above
[12:26:59] <jynus>	 policies should be different, etc.
[12:27:11] <jynus>	 although that was mostly all known
[12:27:26] <jynus>	 the important thing is that I was able to recover month-old backups
[12:27:57] <jynus>	 do you want to give the dbstores a last look
[12:28:19] <jynus>	 or did you do it already, or something else?
[12:31:30] <marostegui>	 no, I already did
[12:31:49] <marostegui>	 nuke them
[12:32:30] <jynus>	 thanks, sorry to ask many times
[12:32:45] <jynus>	 one is never sure of potentially distructive things
[12:34:38] <jynus>	 I saw a backup of db2037 on dbprov2001
[12:34:55] <jynus>	 It doesn't bother me, but is it ok if I put a calendar entry for its deletion?
[12:39:00] <marostegui>	 sure
[12:39:04] <marostegui>	 I will do that
[12:39:12] <jynus>	 1 month? 3 months?
[12:39:18] <jynus>	 ok, you handle it
[12:39:34] <jynus>	 I just want to keep things cleaner on the new servers
[12:39:34] <marostegui>	 I will give it 3 months yeah
[12:45:58] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10jcrespo) The hit rate seems back to normal, so I am guessing in a couple of weeks this could get resolved?
[12:46:59] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) >>! In T210725#5236616, @jcrespo wrote: > The hit rate seems back to normal, so I am guessing in a couple...
[12:48:00] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10jcrespo) > there is still one key to be changed  That is what I meant with "resolved" :-P.
[12:49:11] <jynus>	 wow september, but the year just started!
[12:50:20] <marostegui>	 that's 3 months :p
[12:50:26] <jynus>	 I know
[12:50:45] <marostegui>	 Of course if we run into disk space issues we can kill it ealier
[12:51:19] <jynus>	 the recovery is finishing, I have pending to remove those now
[12:51:39] <marostegui>	 bye bye dbstores....
[12:51:43] <marostegui>	 so long!
[13:00:07] <wikibugs>	 10DBA: Decommission db2042 - https://phabricator.wikimedia.org/T225090 (10Marostegui)
[13:01:33] <wikibugs>	 10DBA: BBU issues on codfw - https://phabricator.wikimedia.org/T214264 (10Marostegui)
[13:01:54] <jynus>	 I am going to start a new codfw backup into es2002,3 of es2,3
[13:02:05] <wikibugs>	 10DBA: Decommission db2042 - https://phabricator.wikimedia.org/T225090 (10Marostegui)
[13:02:21] <jynus>	 you will see lots of activity, it should take I think 12 hours or 1 day
[13:02:50] <wikibugs>	 10DBA: Decommission db2042 - https://phabricator.wikimedia.org/T225090 (10Marostegui)
[13:02:51] <marostegui>	 great
[13:02:55] <marostegui>	 thanks for the heads up :)
[13:03:06] <wikibugs>	 10DBA: Decommission db2042 - https://phabricator.wikimedia.org/T225090 (10Marostegui) p:05Triage→03Normal
[13:08:18] <wikibugs>	 10DBA: Decommission old coredb machines (<=db2042) - https://phabricator.wikimedia.org/T221533 (10Marostegui)
[13:18:20] <jynus>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=es2&var-shard=es3&var-role=All&from=1559729894252&to=1559740694253
[13:18:40] <jynus>	 it may have caused some temporary impact on replication performance on codfw
[13:18:45] <jynus>	 but I see no lag
[13:19:05] <jynus>	 lots of rows read of course, as expected
[13:19:11] <marostegui>	 yeah
[13:19:17] <marostegui>	 I am curious to see how long it takes
[13:19:41] <jynus>	 "but I see no lag" meaning "no further ongoing lag issues after a spike"
[13:20:29] <jynus>	 speed won't be great as those are pure HD hosts
[13:20:40] <jynus>	 and not warmed at all
[14:21:37] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo) a:05jcrespo→03Cmjohnson @robh @Cmjohnson These 3 hosts are ready to be decommissioned. Alerts have been disabled, roles deleted (it is still a spare role),...
[14:22:08] <cmjohnson1>	 Good  afternoon!  db1091...i do have a spare bbu but that spare has been helpful the last year or so. HP is slow to send out the batteries, they can take days to get because of their slow response time and then having to ship batteries via ground transportation only.   If I use it for this server than I am not able to quickly change out the bbu on something that may be more important in the future.   The call
[14:22:08] <cmjohnson1>	 is yours since you have the most BBU issues. 
[14:22:19] <marostegui>	 cmjohnson1: you have a spare BBU??
[14:22:26] <marostegui>	 niiiiiiice
[14:22:27] <cmjohnson1>	 i do but see above
[14:23:20] <marostegui>	 cmjohnson1: Yeah, I see, I think we do need it for this host, as it is one of the ones that support most of the weight in s4 (commonswiki) which is one of the biggest wiksi
[14:23:42] <marostegui>	 cmjohnson1: we might get 2 extra hosts at the end of q1 if analytics are able to free them up, but for now I think we do need db1091 in service
[14:23:59] <cmjohnson1>	 okay, works for me I will get to it today...can you leave it down. 
[14:24:09] <marostegui>	 I will power it off for you yep
[14:24:55] <marostegui>	 cmjohnson1: db1091 is now poweredoff, thank you so much
[14:25:22] <cmjohnson1>	 YW
[14:25:35] <cmjohnson1>	 I m pasting these comments in the task 
[14:25:47] <marostegui>	 great
[14:25:48] <marostegui>	 thanks
[14:26:35] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo)
[14:26:46] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Cmjohnson) Good  afternoon!  db1091...i do have a spare bbu but that spare has been helpful the last year or so. HP is slow to send out the batteries, they can take days to get because of their slow response time...
[14:27:37] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo) a:05Cmjohnson→03RobH
[14:28:40] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo)
[14:29:00] <wikibugs>	 10DBA: BBU issues on codfw - https://phabricator.wikimedia.org/T214264 (10Marostegui)
[14:29:27] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo)
[14:30:14] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10Marostegui)
[14:30:23] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo) I've marked the hosts as "decomissioning", the template and the wiki seem to be outdated and unclear what to do?
[15:59:00] <wikibugs>	 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10jcrespo) I've moved them to active as per volans' advice.
[17:10:24] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Cmjohnson) 05Open→03Resolved The bbu has been replaced.
[17:31:42] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1091 crashed - https://phabricator.wikimedia.org/T225060 (10Marostegui) Thank you so much @Cmjohnson  I can see the battery now: `    Cache Backup Power Source: Batteries    Battery/Capacitor Count: 1    Battery/Capacitor Status: OK `  Next steps I will take: - Start MySQL...