[02:34:46] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Patch-For-Review, 10Schema-change: clean up timestamp database fields - https://phabricator.wikimedia.org/T42626#3791135 (10Krinkle) Looks like a schema change patch exists at <https://gerrit.wikimedia.org/r/110949>.  Unsure about its status and/or opinion from DBAs, and in r...
[06:20:06] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3791378 (10Marostegui) Hey Anomie,   No problen. I only finished dbstore1001 and dbstore1002 for s2. So I am happy to start with s3...
[06:23:33] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3791381 (10Marostegui)
[08:06:10] <jynus>	 why reimage db1099?
[08:06:30] <marostegui>	 to convert it to multi-instance
[08:06:35] <marostegui>	 as it was the plan
[08:07:12] <jynus>	 yes, but reimage? Is it jessie?
[08:07:30] <marostegui>	 We are going for stretch+10.1 for the multi-instance
[08:07:53] <jynus>	 cannot it be upgraded in place and keep the existing data?
[08:08:14] <marostegui>	 I am keeping the existing data by copying it to dbstore1001 and then back
[08:08:17] <marostegui>	 I prefer a full reimage
[08:08:25] <jynus>	 ok, you are doing the extra work
[08:08:32] <jynus>	 so nothing to say about it
[08:08:45] <marostegui>	 I prefer a full reimage if possible
[08:12:33] <jynus>	 :-)
[08:13:10] <marostegui>	 Don't really get the meaning of: "so nothing to say about it" ;)
[08:13:25] <jynus>	 that I will not complain
[08:13:35] <marostegui>	 Ah ok ok
[08:17:05] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3791486 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db1099.eqiad.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017112...
[08:20:09] <jynus>	 heads up for https://gerrit.wikimedia.org/r/#/c/393723/
[08:34:08] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3791572 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1099.eqiad.wmnet'] ```  and were **ALL** successful.
[08:41:18] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3791593 (10Marostegui) The following wikis have to be excluded from the ALTERs in s3, as they are new and were already created with...
[08:58:54] <elukey>	 hello people
[08:59:16] <elukey>	 I did some last minute sanity checks, the log db on dbstore1002 can be dropped
[08:59:29] <marostegui>	 \o/
[08:59:33] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3791657 (10Marostegui) I would appreciate a review for: https://gerrit.wikimedia.org/r/#/c/393725/ @jcrespo @Anomie  There is still...
[08:59:50] <marostegui>	 elukey: you want to do it yourself or you want me to?
[09:00:37] <elukey>	 marostegui: if you have time that'd be great, just to be sure that I don't mess with anything
[09:00:48] <marostegui>	 ok
[09:00:52] <elukey>	 I checked 'show full processlist \G' and processes running
[09:01:00] <elukey>	 yesterday I should have removed all the eventlogging stuff
[09:01:11] <elukey>	 but if you want to triple check it would be great :)
[09:01:19] <marostegui>	 ok
[09:01:20] <marostegui>	 let me see
[09:02:26] <elukey>	 ah I think we can also merge https://gerrit.wikimedia.org/r/#/c/393597/1 ?
[09:03:08] <marostegui>	 did youn run a puppet compiler for db1047, db1046 and dbstore1002 for that change just in case?
[09:03:41] <marostegui>	 ok, let's drop log from dbstore1002 then?
[09:04:10] <elukey>	 marostegui: going to do it now (the pcc)
[09:04:16] <marostegui>	 good
[09:04:17] <elukey>	 yeppa, let's drop it! 
[09:04:18] <elukey>	 \o/
[09:04:47] <marostegui>	 root@DBSTORE[(none)]> set session sql_log_bin=0;
[09:04:47] <marostegui>	 Query OK, 0 rows affected (0.00 sec)<
[09:05:11] <marostegui>	 drop database if exists log;
[09:05:11] <marostegui>	 ok?
[09:05:27] <marostegui>	 looks good?
[09:05:38] <elukey>	 yep
[09:05:43] <marostegui>	 there we go
[09:05:51] <marostegui>	 running
[09:06:02] <marostegui>	 root@DBSTORE[(none)]> drop database if exists log;
[09:06:03] <marostegui>	 Query OK, 379 rows affected (9.01 sec)
[09:06:42] <elukey>	 \o/
[09:07:07] <wikibugs>	 10DBA, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3791689 (10Marostegui) ``` root@dbstore1002:~#  mysql --skip-ssl Welcome to the MariaDB monitor.  Commands end with ; or \g. Your MariaDB...
[09:07:12] <elukey>	 marostegui: https://puppet-compiler.wmflabs.org/compiler02/9003/
[09:07:15] <elukey>	 seems good
[09:07:16] <jynus>	 https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=dbstore1002&var-network=eth0
[09:07:21] <jynus>	 no space gained yet
[09:07:28] <jynus>	 now
[09:07:30] <marostegui>	 I do see space
[09:07:31] <marostegui>	 yeah
[09:07:44] <elukey>	 \o/
[09:08:00] <marostegui>	 elukey: what about db1046?
[09:08:07] <marostegui>	 sorry to be a pain XD
[09:08:16] <elukey>	 ahahah sure I can run it now
[09:09:54] <elukey>	 marostegui: noop - https://puppet-compiler.wmflabs.org/compiler02/9004/
[09:11:10] <elukey>	 ah last thing, I also completed https://gerrit.wikimedia.org/r/#/c/393220/3/modules/role/files/prometheus/mysql-misc_eqiad.yaml
[09:11:19] <elukey>	 that was missing s/db1047/db1108
[09:11:53] <jynus>	 cool
[09:13:14] <elukey>	 it was a long road people but with your help we made it, thanks a lot! 
[09:14:43] <wikibugs>	 10DBA, 10Performance-Team, 10MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), 10Patch-For-Review, 10Wikimedia-log-errors: Frequent "Wikimedia\\Rdbms\\DatabaseMysqlBase::lock failed to acquire lock" errors on WMF mediawiki logs - https://phabricator.wikimedia.org/T180793#3791691 (10jcrespo)...
[09:40:13] <jynus>	 marostegui: one thing we may have to change is how we do schema changes
[09:40:31] <marostegui>	 what do you mean?
[09:40:35] <jynus>	 the order for the next ones mybe should be vertically 
[09:41:00] <jynus>	 s3 codfw -> s3 eqiad -> s2 codfw -> s2 eqiad -> etc
[09:41:18] <marostegui>	 that is how we do it now, right?
[09:41:36] <jynus>	 I think we do all of codfw
[09:41:42] <jynus>	 and then all of eqiad
[09:41:47] <marostegui>	 yes
[09:41:57] <marostegui>	 I am not understanding what you mean I think :)
[09:41:59] <jynus>	 I mean whole shards at a time, on both dcs
[09:42:05] <marostegui>	 Aaaaaah
[09:42:09] <marostegui>	 Right right
[09:42:14] <jynus>	 we do not start with another shard until one is complete
[09:42:17] <marostegui>	 yeah
[09:56:59] <jynus>	 I am going to move db1095:3315 below db1082
[09:57:31] <marostegui>	 cool!
[09:59:01] <jynus>	 then I will stop db1082 and db1087 in sync and create the link to s8 and the filters
[10:00:47] <wikibugs>	 10DBA, 10Data-Services: Provide a new s8 master for sanitarium - https://phabricator.wikimedia.org/T177274#3791741 (10Marostegui)
[10:00:50] <jynus>	 correction, I am going to move db1095:3306, default-master-connection=s5
[10:01:25] <marostegui>	 hehe that is what I understood from your comment by saying 3315, I assumed you meant s5 :)
[10:05:53] <jynus>	 I am not sure I can stop db1082 and db1087 in sync easily
[10:07:07] <marostegui>	 oh right they are replicating from different masters
[10:07:29] <marostegui>	 maybe move them to the same level, do the change and then move one of them back to s8?
[10:07:44] <jynus>	 marostegui: cannot
[10:08:06] <jynus>	 db1095 needs the filters in place
[10:09:04] <jynus>	 I will stop the servers, replicate until a heartbeat and sync them like that
[10:09:20] <marostegui>	 Ah
[10:09:22] <marostegui>	 I see what you mean
[10:09:39] <marostegui>	 they have different hearbeats?
[10:09:43] <jynus>	 filters would be, for s5: existing + wikidatawiki.%,heartbeat,%
[10:09:57] <jynus>	 for s8: existing + dewiki.%
[10:10:03] <jynus>	 that is on ignore
[10:10:11] <marostegui>	 yeah, I was going to ask :)
[10:10:15] <marostegui>	 yes, that makes sense
[10:10:30] <jynus>	 so s8 replicates the heartbeat for the 2
[10:10:47] <jynus>	 which is not ideal because if s5 breaks, 
[10:10:59] <marostegui>	 yes, but that will allow you to do the change you were sayin
[10:10:59] <jynus>	 well, s8 will not replicate anyway
[10:11:02] <marostegui>	 g
[10:11:23] <jynus>	 well, db1082 can break and not db1087
[10:12:09] <jynus>	 on the other side, when db1087 stops replicating s5
[10:12:23] <jynus>	 heartbeat will not be updated on failover
[10:13:01] <jynus>	 and if I make any step wrong, I trash 4 servers
[10:13:53] <marostegui>	 should we maybe put all the steps in a etherpad and review them carefully?
[10:14:03] <marostegui>	 or if you are confident, no need :)
[10:18:07] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3791768 (10Marostegui) db1099:3318 is now replicating
[10:18:43] <jynus>	 I honestly would prefer not to setup anything before failover
[10:19:08] <jynus>	 on switchover, we have perfectly gathered coordinates
[10:19:28] <jynus>	 and if anything happens, a reboot, etc. we may lose filters
[10:19:36] <marostegui>	 that's true
[10:19:38] <jynus>	 I prefer to replicate on s5
[10:19:47] <jynus>	 and with time
[10:20:04] <marostegui>	 and during the failover everytning will be stopped so it is just easier to do the change
[10:20:07] <jynus>	 we setup s8 once s5 actually stops replicating it
[10:20:09] <jynus>	 with no filters
[10:20:17] <jynus>	 marostegui: yes
[10:20:32] <jynus>	 but even if we do it later, it will be easier, I think
[10:20:49] <jynus>	 I do not like adding complexity
[10:21:00] <jynus>	 if it doesn't remove it later
[10:21:04] <jynus>	 and this doesn't
[10:21:20] <marostegui>	 yeah
[10:21:28] <marostegui>	 and the change will not add a lot more time during the failover
[10:21:32] <jynus>	 on swithover there will be no filter
[10:21:44] <jynus>	 just a new connection
[10:21:47] <marostegui>	 exactly
[10:22:45] <jynus>	 I would prefer to take the time to separate s5 and s8 to separate instances
[10:22:51] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3791773 (10Marostegui)
[10:24:32] <marostegui>	 to create one connection per shard you mean?
[10:25:34] <jynus>	 divide db1095 into several instances, as we plan to do
[10:25:53] <marostegui>	 ah yes
[10:26:16] <marostegui>	 Oh I just remembered that we have to move db1095:s3 under db1072
[10:26:18] <marostegui>	 I will try to do that this week
[10:26:21] <jynus>	 although that also cannot happen before the split
[10:27:19] <jynus>	 I can do that if db1072 is row already
[10:27:50] <marostegui>	 yeah, it is row, although I would double check the binlog instead
[10:27:54] <marostegui>	 to be 100% sure
[10:27:56] <marostegui>	 but it should be
[10:28:14] <marostegui>	 s/instead/itself
[10:28:26] <jynus>	 yes, it is
[10:28:30] <jynus>	 at least the last log
[10:32:11] <jynus>	 can I depool db1044- indefinitely?
[10:33:03] <marostegui>	 yeah!
[10:33:29] <marostegui>	 once db1095 is moved under db1072 let's leave it running for a week and then decom db1044 I would suggest
[10:36:34] <jynus>	 by stopping db1044, it is much more easier
[10:36:56] <jynus>	 because I cannot replicate directly from master- technically, I can, but I will not
[10:37:11] <marostegui>	 yeah, just stop them in sync and voila
[10:37:25] * marostegui out for errand+early lunch
[10:37:31] <jynus>	 actually not, I cannot stop db1072
[10:37:36] <jynus>	 but it is easier
[10:41:33] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791841 (10Magnus)
[10:44:02] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791854 (10Magnus) Addendum: ``` MariaDB [wikidatawiki_p]> select @@hostname; +------------+ | @@hostname | +------------+ | labsdb1003 | +------------+ 1 row in set (0.00 sec) ```
[10:47:37] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791841 (10jcrespo) @Magnus -unless you can confirm the same happens on the new hosts (labsdb1009, 10 or 11), I will close this as "won't fix"- new labsdb hosts should be used, and the old ones are o...
[10:56:50] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791908 (10Magnus) "sql" is the recommended shell command to connect to the replicas on Toolforge, correct?  "sql wikidata" connected me to labsdb1003.  I do expect the default settings to give me so...
[10:58:55] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791911 (10Magnus) OK:  ``` MariaDB [wikidatawiki_p]> select @@hostname; +------------+ | @@hostname | +------------+ | labsdb1011 | +------------+ 1 row in set (0.01 sec)  MariaDB [wikidatawiki_p]>...
[11:15:41] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791960 (10jcrespo) > I do expect the default settings to give me something useful. Is that too much to ask?  That is certainly a bug, but it was not the one originally reported "Toolforge Wikidata d...
[11:21:42] <wikibugs>	 10DBA, 10Toolforge: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486#3791989 (10jcrespo) Production masters provide the same result, empty sets-   ``` root@db1070[(none)]> use wikidatawiki Database changed root@db1070[wikidatawiki]> SELECT * FROM wb_terms where term_f...
[11:37:37] <jynus>	 hoo, Amir1 if you have one second, I have some complains about missing rows on wikidata
[11:37:52] <jynus>	 T181486
[11:37:53] <stashbot>	 T181486: Toolforge Wikidata database replica corruption? - https://phabricator.wikimedia.org/T181486
[11:38:01] <jynus>	 it could be just a misunderstanding
[11:38:28] <jynus>	 but I do not know much about the table to say if something is wrong
[11:49:37] <Amir1>	 jynus: I'm around
[11:49:41] <Amir1>	 let me see
[11:57:25] <Amir1>	 jynus: Added my notes there, hope that's helpful 
[11:57:42] <jynus>	 cool
[11:57:44] <jynus>	 thanks
[11:57:59] <jynus>	 can that happen because jobs failing?
[11:58:07] <jynus>	 or during periods of read only?
[11:59:43] <jynus>	 I will let you decide if you want to resolve or followup either you- or someone else
[12:12:18] <Amir1>	 jynus: probably because of readonly or stuff like that, it can happen and there should some sort of safety measures 
[12:12:27] <Amir1>	 I will talk to our PM to see what can be done 
[13:48:16] <wikibugs>	 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3731402 (10chasemp) Thanks @jcrespo for explaining.  Meta point I see here that we keep coming back to and that any variance from has created large hurdles is `Replica databases are sup...
[14:28:58] <wikibugs>	 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3792548 (10jcrespo) Yes, `CREATE TEMPORARY TABLES` can create lag on the replicas for long running queries (locking).
[14:31:24] <jynus>	 https://tendril.wikimedia.org/host/view/db2039.codfw.wmnet/3306
[14:31:28] <jynus>	 See replication lag
[14:31:36] <jynus>	 also dbstore1002 broke again
[14:31:39] <marostegui>	 yeah
[14:31:45] <marostegui>	 is that the table that also broke last week, no?
[14:32:10] <jynus>	 all those *link tables are not very reliable on tokudb slaves
[14:32:30] <jynus>	 do you want me to fix it and reimport it?
[14:32:59] <jynus>	 I am not sure if it is worth reimport it, as we should reload the whole thing
[14:35:28] <marostegui>	 I was thinking maybe we should reimport in innodb?
[14:35:30] <marostegui>	 those tables that fail?
[14:36:34] <jynus>	 nah, the errors is old
[14:36:39] <jynus>	 it is not because it is tokudb
[14:37:07] <jynus>	 but because crashing/not well maintained
[14:37:13] <jynus>	 both 1001 and 1002 are unreliable
[14:37:18] <marostegui>	 then I guess it is easier to fix it thatn wasting time reimporting
[14:37:54] <jynus>	 well, first it has to be fixed no matter way
[14:37:56] <jynus>	 *what
[14:38:06] <jynus>	 because we need it to replicate
[14:38:38] <marostegui>	 yes
[14:44:38] <jynus>	 it was a single row out of a set of 20
[14:45:05] <jynus>	 so, do I leave db1044 replicating or stopped, as it is now?
[14:45:06] <marostegui>	 Pretty much like past week
[14:45:24] <marostegui>	 Sorry to be nitpicking on the patch, it is just a mania I have
[14:45:32] <jynus>	 I don't care
[14:45:49] <jynus>	 as in, tell me which way you prefer
[14:46:12] <marostegui>	 I would leaving replicating
[14:46:16] <jynus>	 ok
[14:46:31] <marostegui>	 But I am not saying we should do it my way :)
[14:47:00] <jynus>	 oh, no, we are doing it your way- which means you will be decommissioning it :-)
[14:47:05] <marostegui>	 haha
[14:48:41] <jynus>	 I will merge now https://gerrit.wikimedia.org/r/393755
[14:49:32] <marostegui>	 go for it
[14:54:28] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#3792646 (10jcrespo)
[14:54:42] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#2714228 (10jcrespo) a:03Marostegui
[15:38:37] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 10 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3792739 (10Addshore) @Legoktm any chance you can take another look at this? :)
[15:47:21] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 11 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3792759 (10Addshore)
[16:19:48] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3792931 (10Marostegui) db1099:3311 is now replicating
[16:19:59] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3792932 (10Marostegui)
[16:27:23] <jynus>	  not sure if still around, but as we may have time, should we get rid of the extra s3 multi-instance server?
[16:27:41] <marostegui>	 o/
[16:27:48] <jynus>	 not now
[16:27:57] <jynus>	 to put on the TODO ehterpad
[16:28:03] <marostegui>	 we do not have any multi-instance on s3
[16:28:07] <marostegui>	 on codfw yes
[16:28:09] <marostegui>	 en eqiad no
[16:28:11] <jynus>	 on codgge
[16:28:13] <jynus>	 arg
[16:28:15] <jynus>	 codfw
[16:28:15] <marostegui>	 haha
[16:28:16] <marostegui>	 yeah
[16:28:20] <marostegui>	 agreed
[16:28:30] <jynus>	 I would like to have the least amount of servers
[16:28:43] <jynus>	 with redundancy
[16:28:47] <marostegui>	 we can actually get rid of both
[16:28:49] <jynus>	 given the current situation
[16:28:55] <jynus>	 well, I mean 1 physical
[16:28:57] <jynus>	 2 instances
[16:29:09] <jynus>	 basically, merging the 2 with s3
[16:29:26] <jynus>	 85 and 92, I think
[16:29:31] <marostegui>	 I would leave it like eqiad
[16:29:40] <jynus>	 yes, that is the idea
[16:30:00] <jynus>	 well, not sure if the pairs are exactly the same
[16:30:08] <jynus>	 but we can check, and set it equaly
[16:30:23] <marostegui>	 No, I mean, s3, only with 2 big ones, like eqiad
[16:30:30] <jynus>	 yes
[16:30:36] <jynus>	 taht is the idea
[16:30:42] <jynus>	 well, 3 big ones
[16:30:48] <jynus>	 with the master
[16:30:51] <jynus>	 and a slow
[16:30:57] <marostegui>	 yeap, didn't count the master 
[16:31:06] <jynus>	 not sure if sizes match fully
[16:31:12] <jynus>	 because we have more servers
[16:31:17] <jynus>	 but less powerful ones
[16:31:26] <jynus>	 I will add it to the todo
[16:31:36] <jynus>	 but I will do the eqiad config first
[16:31:47] <jynus>	 I want to clean up eqiad.php
[16:32:00] <jynus>	 it is a pain right now when there is an ongoing issue
[16:32:05] <marostegui>	 yeah, it needs some big clean up
[16:32:12] <marostegui>	 actually, s3 is the only one that looks clean XD
[16:32:45] <jynus>	 he he
[16:32:47] <jynus>	 yeah
[16:32:53] <jynus>	 that is why I thought about this
[16:32:59] <jynus>	 I will update etherpad
[16:33:02] <marostegui>	 good!
[16:33:04] <marostegui>	 thanks
[16:33:05] <jynus>	 and give a look at the tickets
[17:22:33] <AaronSchulz>	 https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/ is interesting in indeed. Though that reservation example is a bad one in really IMO (e.g. user workflow with multiple HTTP requests mapping to a long-running DB transaction, heh).
[17:22:49] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1051 - https://phabricator.wikimedia.org/T181345#3793300 (10Cmjohnson) Replaced disk 3 and it's rebuilding  Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Rebuild Firmware state: Online, Spun...
[17:24:07] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1051 - https://phabricator.wikimedia.org/T181345#3793321 (10Marostegui) Thanks!
[19:22:53] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1051 - https://phabricator.wikimedia.org/T181345#3794049 (10Marostegui) 05Open>03Resolved a:03Cmjohnson All good - thanks Chris!  ``` root@db1051:~#  megacli -LDPDInfo -aAll  Adapter #0  Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name...
[19:29:46] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T181266#3794066 (10Papaul) @Marostegui Tracking information shows 10:30am CT as delivering time and it is almost 2pm. I contact UPS they let me know that due to the pass holidays the package will not be delivered un...
[19:46:43] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T181266#3794136 (10Marostegui) No problem at all! Thanks for the heads up!
[19:50:29] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T181266#3794148 (10Papaul) a:05Papaul>03Marostegui @Marostegui it looks like UPS give me wrong timing i got the park. Disk replacement complete