[03:57:09] 10DBA, 10CheckUser, 10Patch-For-Review: The "show ip" action should also provide a distinct list of user-agents for each IP - https://phabricator.wikimedia.org/T170508#3999450 (10Huji) @jcrespo I have responded to your request there, and just resolved the merge conflict too. Can you please take a look? [03:57:56] 10DBA, 10CheckUser, 10Patch-For-Review: Create index for cu_agents in cu_changes table - https://phabricator.wikimedia.org/T147894#3999451 (10Huji) @jcrespo can you please take a look at this as well? [07:21:41] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3999486 (10Marostegui) ``` mysql:root@localhost [(none)]> show slave 's7' status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event... [07:25:56] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999488 (10Marostegui) p:05Triage>03High This is s4 primary master - please replace the disk as soon as you can. Thanks! [07:36:32] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3999493 (10Marostegui) Replication is now flowing labsdb1010 for s7. [07:39:36] 10DBA, 10CheckUser, 10Patch-For-Review: The "show ip" action should also provide a distinct list of user-agents for each IP - https://phabricator.wikimedia.org/T170508#3433923 (10Marostegui) Hey @Huji - we, DBAs, are a bit overwhelmed lately with lots of unexpected fires and requests, so it might take someti... [07:39:42] 10DBA, 10CheckUser, 10Patch-For-Review: Create index for cu_agents in cu_changes table - https://phabricator.wikimedia.org/T147894#2707437 (10Marostegui) Hey @Huji - we, DBAs, are a bit overwhelmed lately with lots of unexpected fires and requests, so it might take sometime until we can have a proper look at... [07:45:10] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999509 (10Marostegui) a:03Cmjohnson [10:49:38] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3999591 (10jcrespo) This is just an idea, but maybe we could load balance for some servers not only if the host is down, but also if it has replag > X seconds. This is very very easy work. If you think it is a good idea, I c... [11:01:55] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3999592 (10Marostegui) >>! In T186579#3999591, @jcrespo wrote: > This is just an idea, but maybe we could load balance for some servers not only if the host is down, but also if it has replag > X seconds. This is very very e... [11:02:00] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999213 (10jcrespo) @Cmjohnson Please be extra careful here, there are 2 degraded disks here, but we want to change first _only_ the one shown on the list up there. Once we stop being in non-redundant mode,... [11:08:31] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3999598 (10jcrespo) A good load balancer, like the one we use for mediawiki or varnish doesn't allow depooling more than X hosts or a % of them. In this case, if the 2 are degraded, it should send queries to the least behind... [11:30:47] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999608 (10Marostegui) I believe only the one marked as failed should be blinking in a different colour. The other one only shows errors as far as the report goes, but yeah, better be careful if there are tw... [12:18:53] 10DBA: memcached on db1011/tendril not puppetised - https://phabricator.wikimedia.org/T133906#2248522 (10Marostegui) I would say we can close this as declined or invalid as we have migrated to db1115 and we don't even have memcached installed there and as far as I know we are not even considering doing it so. `... [12:26:33] 10DBA: memcached on db1011/tendril not puppetised - https://phabricator.wikimedia.org/T133906#3999650 (10jcrespo) Let's check if the frontend code depends on memcached at dbmonitor, maybe there is code failing and we don't even notice; or it maybe it was never implemented.. [12:36:23] 10DBA: memcached on db1011/tendril not puppetised - https://phabricator.wikimedia.org/T133906#3999651 (10jcrespo) Memcached is expected, but on frontend, not on backend: ```name=dbmonitor1001,lang=php $_ENV['mc_host'] = '127.0.0.1'; $_ENV['mc_port'] = 11211; ``` [12:37:15] 10DBA: install and puppetize memcached on dbmonitor hosts (tendril frontends) - https://phabricator.wikimedia.org/T133906#3999652 (10jcrespo) [15:02:43] 10DBA, 10CheckUser, 10Patch-For-Review: The "show ip" action should also provide a distinct list of user-agents for each IP - https://phabricator.wikimedia.org/T170508#3999748 (10Huji) This is not super urgent; we had forgotten to add #DBA to it at the get go, so we are at fault. Get to it as you can.