[05:15:56] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2053 - https://phabricator.wikimedia.org/T203623 (10Marostegui) p:05Triage>03Normal a:03Papaul Can we get a new disk here @papaul?  Thanks!
[05:44:36] <wikibugs>	 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) I have been syncing-up with @ayounsi about the scheduled network maintenance and the switches issues. So far they are still doing some tests (T201145) and they should know a bit more how to proceed further in a few day...
[06:05:10] <jynus>	 check https://gerrit.wikimedia.org/r/#/c/457847/ one last time and I will deploy it now
[06:52:21] <elukey>	 hello :)
[06:52:37] <elukey>	 jynus: I am reviewing https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454291/ but I don't have much context on it
[06:53:09] <elukey>	 this is only to remove non-system mysql users?
[06:53:22] <jynus>	 yes
[06:53:39] <jynus>	 I know analytics have some non-core mysql hosts
[06:53:54] <jynus>	 I want to make sure this doesn't break that installation
[06:54:04] <jynus>	 aka all your mysql users are system users
[06:54:04] <elukey>	 thanks a lot for the heads up, going to check
[06:54:29] <jynus>	 it should not happen if you are using jessie or higher
[06:54:35] <jynus>	 but better safe than sorry
[06:55:18] <elukey>	 so the check is basically to id mysql on the hosts in which we run non-core dbs and make sure that nothing is > 999
[06:55:50] <jynus>	 the actual id may vary depending on the os
[06:55:59] <jynus>	 for system users, I mean
[06:56:03] <elukey>	 yep yep
[06:56:26] <jynus>	 check /etc/adduser.conf
[06:56:45] <jynus>	 but yes, in debian latest versions I think it starts at 1000
[06:56:55] <jynus>	 grep mysql /etc/passwd
[06:57:37] <jynus>	 sadly, mysql users used to be created as a non-system user
[06:57:43] <jynus>	 years ago
[06:57:50] <jynus>	 so some host may have survived
[06:58:19] <jynus>	 I checked all db* hosts, but could be other usages
[06:58:51] <elukey>	 +1
[06:59:17] <elukey>	 we have dbs on bohrium (piwik) and analytics1003 (for hadoop services)
[06:59:20] <elukey>	 all good afaics
[06:59:51] <jynus>	 thanks, I hoped so
[06:59:55] <jynus>	 I will deploy this soon
[07:03:46] <elukey>	 about https://phabricator.wikimedia.org/P7516, are those things in wikitech or presentations that you guys already gave to new opsens?
[07:14:40] <jynus>	 [09:07] <jynus> some are on wikitech
[07:14:42] <jynus>	 [09:07] <jynus> some overview is in presentations
[07:14:43] <jynus>	 [09:07] <jynus> but mostly is not
[07:14:45] <jynus>	 [09:07] <jynus> mostly is on puppet
[07:14:46] <jynus>	 [09:08] <jynus> or things a dba should now indepentendly of WMF knwledge
[07:15:50] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10Marostegui) I am happy to merge this task with T161859  if @jcrespo is (as he is the original task creator)
[07:26:16] <volans>	 jynus: in yesterday's test of the ReadOnly message, did you by any chance check it also in other wikis (not english) and notice if it gets translated or not?
[07:27:06] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10jcrespo) why not just make this a child of the other and agree to talk only on a single place? This is mostly for DBA work of data migration, which is...
[07:27:21] <wikibugs>	 10DBA, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10jcrespo)
[07:27:52] <wikibugs>	 10DBA, 10wikitech.wikimedia.org: Move wikitech and labstestwiki to s5 - https://phabricator.wikimedia.org/T167973 (10jcrespo)
[07:29:06] <jynus>	 volans: I only checked dewiki and enwiki
[07:29:21] <jynus>	 however, I didn't check because I have all wikis in english interface
[07:29:56] <jynus>	 and I couldn't change it because you cannot change the preferences in read only
[07:30:10] <jynus>	 you can use uselang=<lang> to test it
[07:30:29] <volans>	 if I use the mwdebug extension and go to it.wikipedia and try to edit should do the same right?
[07:30:56] <jynus>	 if you chose a codfw debug server, yes
[07:31:02] <volans>	 yeah ofc :D
[07:31:17] <volans>	 ok so visualeditor show me a big red box with a long italian message
[07:31:18] <jynus>	 I am guessing it has italian as the default language
[07:31:22] <jynus>	 or you do on your browser
[07:31:38] <volans>	 as a pop notification on the top right
[07:31:44] <jynus>	 visualeditor I think ignores the admin message
[07:31:58] <volans>	 same for text edit, top red banner
[07:32:01] <jynus>	 I would consider that as a bug
[07:32:14] <volans>	 with a long italian message
[07:32:20] <jynus>	 I think the wikitext editor does the right thing
[07:32:29] <jynus>	 show a message from the interface
[07:32:39] <jynus>	 and then say the admin showed the following reason:
[07:32:40] <volans>	 none of those show me the REadOnly message in etcd, but another longer message
[07:32:45] <jynus>	 and show the read only message
[07:32:46] <volans>	 ah right
[07:32:48] <volans>	 in the middle
[07:33:03] <volans>	 yeah sorry pebcak
[07:33:04] <jynus>	 sometimes it can be overriden if the parameter is not set
[07:33:08] <jynus>	 no no
[07:33:08] <volans>	 the etcd message is in both
[07:33:14] <volans>	 in english after the localized message
[07:33:17] <volans>	 so I think it's ok
[07:33:24] <jynus>	 if the translation or the wiki has been localized without the message
[07:33:26] <jynus>	 it can happen
[07:33:31] <jynus>	 but I guess that is not our problem
[07:33:36] <volans>	 ... L'amministratore che impostato il blocco ha fornito questa spiegazione: MediaWiki is in read-only mode for maintenance. Please try again in a few minutes..
[07:33:53] <jynus>	 volans: should we link to meta?
[07:33:58] <jynus>	 for more info
[07:34:29] <volans>	 not sure if links will actually link, I remember some discussion from last year
[07:34:39] <volans>	 and I'm waiting the official message from comms
[07:34:43] <volans>	 that's the one I'll be using
[07:35:01] <volans>	 luckily I put it as a CLI param, so we can change it last minute :D
[07:37:46] <jynus>	 we used to have https://meta.wikimedia.org/wiki/Tech/Server_switch_2017
[07:37:47] <jynus>	 ah, yes
[07:37:47] <jynus>	 on wikitext it should work
[07:37:47] <jynus>	 volans: it is easy to test now, just an etcd deploy away
[07:37:48] <jynus>	 however, the main issue is multi-language support
[07:37:50] <jynus>	 the 2017 message was translated, we don't have that now
[07:37:51] <jynus>	 volans: what happens if you try to log in in read only?
[07:37:51] <jynus>	 does it fail?
[07:40:31] <volans>	 mmmh nope
[07:40:33] <volans>	 I can login
[07:41:08] * volans brb in ~10
[07:54:20] <marostegui>	 I am going to start enabling replication codfw -> eqiad in s1-s8, any obejctions?
[07:54:58] <jynus>	 wait
[07:55:05] <jynus>	 I was about to deploy the codfw change
[07:55:11] <marostegui>	 ok
[07:55:48] <jynus>	 also do the change masters but let me do a double check before start slave- while the change is trivial
[07:55:54] <jynus>	 a simple mistake can be very bad
[07:56:07] <jynus>	 better 2 people checking
[07:56:10] <marostegui>	 yeah, I was planning to do so anyways :)
[07:56:37] <jynus>	 also let me double check the backups
[08:00:54] <marostegui>	 did the partitioning for the s1 codfw hosts finished?
[08:03:59] <jynus>	 marostegui: yes
[08:04:00] <jynus>	 I didn't touch the others, something to fix for another time
[08:04:00] <jynus>	 backups are ok, I can see it on the metadata gathered
[08:04:40] <jynus>	 you can proceed with the replication changes, let's start with x1 or s5
[08:05:09] <marostegui>	 yeah, I was planning to do: s5,s6,s2 to start with
[08:05:31] <marostegui>	 I asked about the partitioning cause I am updating our etherpad
[08:05:33] <marostegui>	 with the pre-things
[08:07:08] <jynus>	 sorry, I forgot about that
[08:08:27] <marostegui>	 jynus: check db1070 show slave status
[08:08:31] <marostegui>	 that is s5
[08:08:37] <jynus>	 I kind of did line 65
[08:08:42] <jynus>	 but I need to do more
[08:09:34] <jynus>	 downtime the replication checks, it may page
[08:09:42] <jynus>	 if we take some time setting it ip
[08:10:14] <marostegui>	 done
[08:10:24] <jynus>	 also log the actions at least once
[08:10:31] <marostegui>	 I logged it
[08:10:39] <jynus>	 sorry, didn't see it
[08:10:45] <jynus>	 too much going on right now :-)
[08:10:48] <marostegui>	 ;)
[08:11:22] <jynus>	 s5 is ok to start
[08:11:29] <marostegui>	 ok, starting it
[08:11:31] <jynus>	 should I check others?
[08:11:38] <marostegui>	 no, I am doing one at the time
[08:11:47] <marostegui>	 I will let you know when you can check the others
[08:11:59] <jynus>	 I think it cannot page
[08:12:02] <marostegui>	 We should also disable GTID on codfw
[08:12:06] <marostegui>	 (on codfw masters)
[08:12:10] <jynus>	 as stopped is only a warning
[08:12:14] <marostegui>	 but let's do that once all the replication is set up
[08:12:23] <jynus>	 gtid on the master?
[08:12:33] <marostegui>	 codfw masters replicate from eqiad, and gtid is enabled
[08:12:34] <jynus>	 it should be disabled by default?
[08:12:42] <jynus>	 maybe we enable it
[08:12:45] <marostegui>	 not sure, we should double check
[08:12:48] <marostegui>	 and if so, disable it
[08:12:50] <jynus>	 sure
[08:12:53] <jynus>	 +1
[08:13:00] <marostegui>	 that is on line 56 so we don't forget
[08:13:43] <marostegui>	 check db1061 (s6 master)
[08:14:43] <jynus>	 looks good
[08:14:48] <marostegui>	 enabling it
[08:15:02] <jynus>	 maybe there is not reason to do the checks?
[08:15:11] <marostegui>	 no, I prefer to get them checked
[08:15:14] <jynus>	 worse case scenario it has to fail immediately
[08:15:22] <marostegui>	 check db1066 (s2 master)
[08:15:26] <jynus>	 unless it is x1->s* or s* -> x1?
[08:15:48] <marostegui>	 We do this once a year, so I prefer to get another pair of eyes, even if we waste 15 minutes of our lifes XD
[08:16:02] <jynus>	 we can automate it!
[08:16:22] <jynus>	 s2 looks good
[08:16:37] <marostegui>	 enabling
[08:16:47] <jynus>	 worst time was bad beacuse it brought down tendril with an infitnite loop
[08:17:29] <jynus>	 good times
[08:17:29] <jynus>	 (first time)
[08:18:03] <marostegui>	 check s1 master db1067
[08:20:01] <jynus>	 db2048 ok
[08:20:13] <marostegui>	 enabling
[08:20:24] <jynus>	 I am checking logs at the same time too
[08:20:56] <marostegui>	 check s3 master db1075
[08:22:46] <jynus>	 it is ok
[08:22:49] <marostegui>	 enabling
[08:23:40] <marostegui>	 check s4 master db1068
[08:24:33] <jynus>	 all good
[08:24:40] <marostegui>	 enabling
[08:24:45] <marostegui>	 and only s7,s8 and x1 pending
[08:24:52] <jynus>	 es2 and 3
[08:24:59] <marostegui>	 yeah, I mean from core
[08:25:07] <jynus>	 es is core :-)
[08:25:13] <jynus>	 :-P
[08:25:17] <marostegui>	 XD
[08:25:19] <marostegui>	 es is es!
[08:26:01] <jynus>	 so I call them "core metadata"
[08:26:07] <marostegui>	 check s7 master db1062
[08:26:08] <jynus>	 and es "core content"
[08:26:29] <marostegui>	 good naming
[08:26:46] <jynus>	 as all litearally implement mariadb::core on puppet
[08:27:19] <jynus>	 s7 good
[08:27:23] <marostegui>	 enabling
[08:28:10] <marostegui>	 check s8 master db1071
[08:28:39] <jynus>	 looks good
[08:28:43] <marostegui>	 enabling
[08:29:32] <marostegui>	 check x1 master db1069
[08:31:10] <volans>	 marostegui: jaime raised an exception while checking :-P
[08:31:24] <marostegui>	 checking overflow
[08:31:45] <jynus>	 db2034 looks good
[08:31:52] <marostegui>	 enabling
[08:32:03] <marostegui>	 going for es2,3 now!
[08:32:13] <jynus>	 did we do x1 or later?
[08:33:10] <marostegui>	 x1 was db1069 :)
[08:33:26] <marostegui>	 the one you just checked XD
[08:33:37] <jynus>	 ok
[08:33:39] <marostegui>	 check es2 master es1015
[08:34:24] <jynus>	 everything ok
[08:34:27] <marostegui>	 ok enabling
[08:35:20] <marostegui>	 check es3 master, last one! es1017
[08:35:46] <marostegui>	 we've got a page
[08:35:48] <marostegui>	 checking that
[08:46:15] <jynus>	 lets continue
[08:46:22] <marostegui>	 yeah
[08:46:28] <marostegui>	 check es3 master es1017
[08:46:32] <marostegui>	 that is the last one
[08:46:57] <jynus>	 we are good
[08:47:20] <marostegui>	 done
[08:47:21] <marostegui>	 we are good then
[08:47:26] <marostegui>	 I will check the GTID 
[08:47:32] <marostegui>	 to see if it is enabled
[08:47:54] <jynus>	 so you only deleted the heartbeat row to start x1?
[08:47:58] <marostegui>	 yeah
[08:48:04] <jynus>	 so we can repool it, se patch
[08:48:10] <marostegui>	 yeah
[08:48:41] <jynus>	 should be disable gtid now?
[08:48:43] <wikibugs>	 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) Replication codfw -> eqiad has been enabled on s1-s8,x1,es2,es3
[08:48:44] <volans>	 I can run the check the switcdc does
[08:48:52] <volans>	 if you want
[08:49:03] <jynus>	 should we wait until next week?
[08:49:11] <marostegui>	 to disable GTID?
[08:49:19] <jynus>	 there is a chance of a master host crashing?
[08:49:23] <marostegui>	 sure, that is a good a idea
[08:49:25] <jynus>	 I am mostly asking, not sure
[08:49:29] <marostegui>	 No, it is a good idea
[08:49:35] <marostegui>	 No benefit of doing it now
[08:49:40] <marostegui>	 I will create a calendar event for tuesday
[08:49:43] <jynus>	 set it on the todo
[08:49:45] <jynus>	 that
[08:49:49] <marostegui>	 yep
[08:49:54] <jynus>	 I think volans' check will still work
[08:50:10] <volans>	 and we always test+modify at will
[08:50:21] <jynus>	 and or we can check its state in any case
[08:51:01] <volans>	 so the check right now worked (was failing yesterday ofc)
[08:51:04] <jynus>	 yeah, the inverse check won't work
[08:51:06] <volans>	 for all core DBs
[08:51:19] <volans>	 what do you mean jynus ?
[08:51:31] <jynus>	 if you check the codfw -> eqiad
[08:51:37] <jynus>	 it will fail
[08:51:40] <volans>	 it's what I just did
[08:51:46] <volans>	 it was failing yesterday, works now
[08:51:53] <marostegui>	 \o/
[08:51:55] <jynus>	 that should not work
[08:52:01] <volans>	 why?
[08:52:06] * volans missing something
[08:52:19] <jynus>	 because the current masters are not using gtid
[08:52:29] <jynus>	 so you should not be able to check its gtid position
[08:52:49] <jynus>	 I guess wait will return inmedietly
[08:53:00] <jynus>	 which is why it works now
[08:53:03] <volans>	 it retuns immediately with 0
[08:53:16] <jynus>	 but it is not checking what we want to check
[08:53:26] <jynus>	 another reason to avoid gtid- missleading info
[08:53:36] <volans>	 and doesn't fail?
[08:53:38] <volans>	 :(
[08:53:58] <volans>	 but it was failing yesterday
[08:54:08] <jynus>	 yeah, beacause it wasn't replicating
[08:54:16] <jynus>	 now it replicates but without gtid
[08:54:36] <jynus>	 when we do a check, please print the debug and send it to me
[08:54:47] <jynus>	 I need to attend joe
[08:54:51] <volans>	 so, what query should I use to check?
[08:55:28] <jynus>	 send me a detailed debug
[08:55:29] <marostegui>	 I am going to deploy your change to repool db1120, jynus
[08:55:35] <jynus>	 please do
[08:55:39] <marostegui>	 :)
[08:58:03] <volans>	 jynus: https://phabricator.wikimedia.org/P7519
[08:58:17] <volans>	 that's x1, the others have the same logic
[08:58:42] <jynus>	 171966572-171966572-297840043,171974681-17197468, 1-198565537,180355159-180355159-17196909,180363268-180363268-40608909
[08:58:47] <jynus>	 yeah, that is bad
[09:01:09] <jynus>	 the replacement, however, is not simple
[09:01:20] <jynus>	 SHOW MASTER STATUS\G
[09:01:39] <jynus>	 and then comparing each column individually
[09:01:49] <jynus>	 or
[09:01:56] <jynus>	 we can check heartbeart
[09:02:18] <volans>	 jynus: but this has changed since last year? (we had GTID everywhere last year?)
[09:02:38] <jynus>	 yes and no
[09:02:47] <jynus>	 we know since last year, gtid is unreliable
[09:02:51] <jynus>	 does that count?
[09:02:59] <volans>	 lol
[09:05:06] <jynus>	 we can do "select ts FROM heartbeat.heartbeat WHERE datacenter = 'eqiad' and shard = 'es3' ORDER BY ts DESC LIMIT 1";
[09:05:40] <jynus>	 on master and replica and to master should be strictly larger than master or retry
[09:06:01] <jynus>	 'eqiad' is the datacenter from
[09:07:00] <jynus>	 do you want me to write the code?
[09:07:25] <volans>	 of it's that it's easy
[09:07:29] <volans>	 we can keep the same logic
[09:07:36] <volans>	 I should be able to send a patch shortly
[09:07:39] <jynus>	 but there is no wait
[09:07:45] <volans>	 we retry
[09:07:47] <jynus>	 we have to implement the wait on code
[09:07:52] <volans>	 @retry
[09:07:52] <_joe_>	 ok so it's easier than it seemed?
[09:08:00] <volans>	 we've a nice decorator for that
[09:08:37] <_joe_>	 nice to hear, I was starting to worry
[09:08:37] <jynus>	 that is likely the solution we are going to do for mediawiki aswell
[09:08:56] <jynus>	 this step anyway is non-critical
[09:09:10] <jynus>	 we could have added a 10 second wait and we would be mostly ok
[09:09:36] <jynus>	 the reason it worked now is because gtid was "contaminated" on both sides
[09:33:26] <volans>	 jynus: strictly > or >= ?
[09:34:12] <jynus>	 >
[09:34:24] <jynus>	 because it only hits every second
[09:34:37] <jynus>	 we need to make sure a whole second is after 
[09:34:51] <volans>	 ok so master in the new DC must have a newer value compared to the one just read from old DC
[09:34:54] <jynus>	 you may want to wait one second on client to avoid retries
[09:34:54] <volans>	 ack
[09:35:10] <jynus>	 and do the checks in parallel
[09:35:19] <jynus>	 to avoid waiting 20 seconds in series
[09:35:37] <jynus>	 so to.heartbeat > from.heartbeat
[09:35:40] <volans>	 ok
[09:35:54] <jynus>	 that will work without read only, but will not really test anything
[09:36:24] <volans>	 cannot do the checks in parallel as of now
[09:36:25] <jynus>	 so it is easy to test right now
[09:36:43] <jynus>	 okso wait 1 second and do the checks when you can
[09:36:53] <jynus>	 in other words
[09:36:59] <volans>	 yeah eyah got it
[09:37:14] <jynus>	 can you gather the from master info quickly
[09:37:23] <jynus>	 and then check quickly
[09:37:38] <jynus>	 from, eg, an array?
[09:37:57] <jynus>	 I can help
[09:38:09] <jynus>	 other thing is if you trust my code
[09:39:21] <jynus>	 marostegui: we can now use the new $mw_primary local variable with multiiinstance
[09:39:30] <marostegui>	 \o/
[09:39:36] <jynus>	 and I can deploy the read checks (but not this week)
[09:39:41] <marostegui>	 yeah
[09:39:42] <jynus>	 *read only
[09:39:51] <marostegui>	 will you also deploy the AUTO change?
[09:40:17] <jynus>	 no, there is no auto
[09:40:26] <jynus>	 the solution is to check on "client"
[09:40:30] <marostegui>	 So this can also move forward: https://phabricator.wikimedia.org/T199124
[09:40:39] <jynus>	 but now it is better
[09:40:47] <jynus>	 as it is etcd controlled
[09:40:55] <jynus>	 not hiera controlled
[09:40:57] <marostegui>	 So we don't need this anymore? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/449711/
[09:41:12] <jynus>	 https://phabricator.wikimedia.org/T199124 has just been resolved
[09:41:19] <marostegui>	 \o/
[09:41:40] <wikibugs>	 10DBA, 10Operations, 10Puppet: Remove all usages of $::mw_primary on puppet - https://phabricator.wikimedia.org/T199124 (10jcrespo) a:05jcrespo>03Joe
[09:42:30] <jynus>	 marostegui: we need still to change the current behaviour
[09:42:39] <jynus>	 everthing is critical
[09:42:47] <jynus>	 to the same way the core is done now
[09:43:19] <marostegui>	 Ah, so basically only need to change: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/449711/8/hieradata/role/common/mariadb/core_multiinstance.yaml
[09:43:25] <marostegui>	 the rest should stay as it is
[09:43:43] <jynus>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/457491/4/modules/role/manifests/mariadb/core.pp
[09:43:50] <jynus>	 ^like this
[09:43:58] <jynus>	 it still needs some changes
[09:44:09] <jynus>	 as I think the latest version is critical=1 for all
[09:44:19] <marostegui>	 yeah
[09:44:42] <jynus>	 but no icinga or script changes
[09:44:48] <marostegui>	 that is great
[09:45:26] <marostegui>	 So what's the plan with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/449742/ ?
[09:45:42] <jynus>	 I will abandon both options
[09:45:57] <jynus>	 joe implemented a sort of better version of
[09:46:26] <marostegui>	 Cool, I have lost track of all the patches, so I will probably need help with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/449711/
[09:47:09] <jynus>	 https://gerrit.wikimedia.org/r/345346
[09:47:33] <_joe_>	 marostegui: I can help I guess, PM me when you need it
[09:47:54] <marostegui>	 Thanks _joe_ will do :)
[09:48:30] <jynus>	 I can do that
[09:48:35] <jynus>	 I kept track
[09:48:45] <marostegui>	 :-) thanks!
[10:11:59] <volans>	 I'm senting a preview of the code while I write the tests
[10:12:19] <volans>	 *sending
[10:13:50] <marostegui>	 we are on a meeting :)
[10:20:10] <volans>	 jynus, marostegui: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/458470
[14:04:20] <wikibugs>	 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423 (10Marostegui) Just for the record. The only hosts we have running with multi-source at the moment are:  - labsdb1009-1011 - dbstore1002 (which will be gone once the new HW is bought an...
[15:48:56] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2053 - https://phabricator.wikimedia.org/T203623 (10Marostegui) Talked to Papaul - this disk will be replaced on Monday, as he is on a different DC! Thanks!
[16:48:29] <volans>	 marostegui, jynus: FYI updated paste with the switchdc check: https://phabricator.wikimedia.org/P7519
[17:18:37] <wikibugs>	 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: HAproxy on dbproxy hosts lack enough logging - https://phabricator.wikimedia.org/T201021 (10jcrespo) 05Resolved>03Open p:05Normal>03High There is now too much logging, or it is not rotated fast enough: logs are consuming 70% of available disk:...
[17:30:19] <wikibugs>	 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: HAproxy on dbproxy hosts lack enough logging - https://phabricator.wikimedia.org/T201021 (10Marostegui) I will play with the different log levels tomorrow to see which is the minimum we can do to still get the requests logged, or at least the failures
[17:39:38] <wikibugs>	 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: HAproxy on dbproxy hosts lack enough logging - https://phabricator.wikimedia.org/T201021 (10Marostegui) I have purged logs from other dbproxies too just to make sure they are ok.
[19:45:27] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Ladsgroup)
[19:45:59] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) p:05Triage>03Normal
[19:48:24] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata-Campsite, 10User-Ladsgroup: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Ladsgroup) a:05Ladsgroup>03None
[23:24:21] <wikibugs>	 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Pine)