[00:16:03] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891821 (10MaxSem)
[00:18:47] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891821 (10MaxSem)
[00:22:17] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891852 (10MaxSem)
[00:31:45] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891867 (10MaxSem)
[07:38:59] <wikibugs>	 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3892272 (10Marostegui)
[07:39:19] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3892276 (10Marostegui)
[07:39:21] <wikibugs>	 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3877254 (10Marostegui) 05Open>03Resolved a:03Marostegui All done
[08:50:12] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3892358 (10jcrespo) The thing is, we already log on SAL whenever we reboot the server- the real coordination is to check for SAL entr...
[11:07:43] <wikibugs>	 10DBA: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#3892707 (10jcrespo)
[11:10:00] <wikibugs>	 10DBA: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#3892723 (10Marostegui) p:05Triage>03Normal
[11:14:00] <wikibugs>	 10DBA: Failover existing eqiad database backup system to this new system - https://phabricator.wikimedia.org/T184697#3892734 (10jcrespo)
[11:14:30] <wikibugs>	 10DBA: Failover existing eqiad database backup system to the new codfw database logical backup system - https://phabricator.wikimedia.org/T184697#3892743 (10jcrespo)
[11:14:45] <wikibugs>	 10DBA: Failover existing eqiad database backup system to the new codfw database logical backup system - https://phabricator.wikimedia.org/T184697#3892744 (10Marostegui) p:05Triage>03Normal
[11:15:18] <wikibugs>	 10DBA: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#3892746 (10jcrespo)
[11:15:20] <wikibugs>	 10DBA: Failover existing eqiad database backup system to the new codfw database logical backup system - https://phabricator.wikimedia.org/T184697#3892745 (10jcrespo)
[11:18:13] <wikibugs>	 10DBA, 10Operations, 10Goal: Generate consistent logical database backups in CODFW - https://phabricator.wikimedia.org/T184699#3892765 (10jcrespo)
[11:18:34] <wikibugs>	 10DBA, 10Operations, 10Goal: Generate consistent logical database backups in CODFW - https://phabricator.wikimedia.org/T184699#3892785 (10Marostegui) p:05Triage>03Normal
[11:20:13] <wikibugs>	 10DBA: Failover existing eqiad database backup system to the new codfw database logical backup system - https://phabricator.wikimedia.org/T184697#3892790 (10jcrespo)
[11:20:15] <wikibugs>	 10DBA: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#3892791 (10jcrespo)
[11:20:22] <wikibugs>	 10DBA, 10Goal: Check data consistency across production shards - https://phabricator.wikimedia.org/T183735#3892792 (10jcrespo)
[11:20:25] <wikibugs>	 10DBA, 10Operations, 10Goal: Generate consistent logical database backups in CODFW - https://phabricator.wikimedia.org/T184699#3892789 (10jcrespo)
[11:21:22] <wikibugs>	 10DBA: Check data consistency across production shards - https://phabricator.wikimedia.org/T183735#3892796 (10Marostegui)
[11:21:36] <wikibugs>	 10DBA: Check data consistency across production shards - https://phabricator.wikimedia.org/T183735#3862188 (10Marostegui)
[11:30:07] <wikibugs>	 10DBA: db1011 possibly faulty BBU - https://phabricator.wikimedia.org/T184401#3892846 (10jcrespo) 05Open>03Resolved a:03Marostegui it is working well--- for now, needs replacement.
[11:31:12] <wikibugs>	 10DBA: Decommission db1011 - https://phabricator.wikimedia.org/T184703#3892852 (10jcrespo)
[11:31:46] <wikibugs>	 10DBA: Decommission db1011 - https://phabricator.wikimedia.org/T184703#3892864 (10jcrespo)
[11:31:49] <wikibugs>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3892865 (10jcrespo)
[11:32:29] <wikibugs>	 10DBA: Decommission db1011 - https://phabricator.wikimedia.org/T184703#3892868 (10Marostegui) p:05Triage>03Normal
[11:32:40] <wikibugs>	 10DBA: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#3892870 (10jcrespo)
[11:33:05] <wikibugs>	 10DBA: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#3892881 (10jcrespo)
[11:33:57] <wikibugs>	 10DBA: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#3892887 (10Marostegui) p:05Triage>03Normal
[11:59:54] <mark>	 hey guys, sorry I missed our meeting :(
[12:03:59] <marostegui>	 hey mark
[12:04:14] <marostegui>	 We were wondering if you could take a look at T184106
[12:06:30] <mark>	 yes
[12:06:37] <mark>	 and if you want we can still do the meeting later today or tomorrow
[12:07:01] <mark>	 i'll get that procurement going
[12:07:08] <marostegui>	 thanks! :)
[12:07:47] <marostegui>	 I think the most "urgent" thing from our side was that ticket, at least that was the thing I had for the meeting, so from my side I am fine :)
[15:03:36] <jynus>	 any scheduled schema changes on db1102 or db1095?
[15:03:49] <marostegui>	 nope
[15:03:53] <marostegui>	 (for this week)
[15:04:21] <jynus>	 I will then take the change to upgrade those
[15:04:29] <marostegui>	 cool!
[15:05:03] <jynus>	 I know it is not very urgent but they end up being blockers of other production hosts
[16:05:28] <marostegui>	 Interesting, I was testing if a replacing a row on a server running RBR, with a slave that doesn't have that row would break replication
[16:05:52] <marostegui>	 I thought it would as the replace actually does a delete, I would expect it to fail as the delete cannot happen on the slave because it is missing a row
[16:05:55] <marostegui>	 Interesting
[16:07:06] <marostegui>	 I thought it would look for the PK of that row, and as it doesn't exist it would complain
[16:09:33] <jynus>	 sorry, I do not understand what you mean- what did you try and happened?
[16:09:39] <marostegui>	 hehe sorry
[16:10:01] <marostegui>	 so while doing all the data drift fixes and all that, all of a sudden I have a doubt and went and did some testing on my own lab
[16:10:05] <marostegui>	 The doubt was
[16:10:15] <jynus>	 give me some rows and sql
[16:10:19] <marostegui>	 hahaha
[16:10:24] <marostegui>	 let me explain first!
[16:10:26] <jynus>	 not formal
[16:10:33] <jynus>	 just the idea
[16:11:09] <marostegui>	 So, if we are running RBR on a host, with slaves, and we issue a REPLACE on that master, what happens if slaves do not have that specific row
[16:11:18] <marostegui>	 I thought they would break replication
[16:11:23] <marostegui>	 As the row is missing
[16:12:17] <jynus>	 what do you get on the binlog?
[16:12:34] <jynus>	 of master and replica?
[16:12:38] <marostegui>	 exactly
[16:12:48] <marostegui>	 a further examination of the binlog reveals it is an insert
[16:13:19] <jynus>	 an insert on both master and slave?
[16:13:25] <marostegui>	 correct
[16:13:27] <marostegui>	 as the row is missing
[16:13:57] <marostegui>	 if the row exists, it is obvioulsly an UPDATE
[16:14:00] <jynus>	 but it would fail the other way
[16:14:04] <jynus>	 ?
[16:14:22] <jynus>	 because the update would fail?
[16:15:03] <jynus>	 I assume there is no concept of INSERT,update,replace on ROW, just the row change, which then gets reinterpreted
[16:15:33] <jynus>	 old row, new row
[16:15:50] <marostegui>	 Yeah, if the row exists on the master and not on the slave
[16:15:50] <marostegui>	 it breaks
[16:15:50] <marostegui>	 (as I expected)
[16:16:20] <jynus>	 I think row can be configured to be more strict
[16:16:22] <marostegui>	 But the funny thing is
[16:16:38] <marostegui>	 let's say you don't have any row
[16:16:51] <marostegui>	 and insert a new row on the master but instead of using INSERT INTO, you just use replace
[16:16:54] <marostegui>	 then it doesn't break
[16:18:17] <jynus>	 I do not see why it is weird
[16:18:24] <jynus>	 it is bad if you have drift
[16:18:30] <jynus>	 because it went undetected
[16:18:51] <jynus>	 but if you think with old row, new row it makes sense
[16:19:36] <marostegui>	 yeah, but if you create a new row on both, master and slave with REPLACE, I would expect that replace to also break replication
[16:19:40] <jynus>	 remember that if you do an update and the row has different initial fields before PK
[16:19:47] <marostegui>	 but obviously not, because the binlog gets an INSERT
[16:19:51] <jynus>	 marostegui:  on STATEMENT
[16:20:12] <jynus>	 actually, I may not have understood that last one
[16:20:25] <marostegui>	 the last test was
[16:20:36] <marostegui>	 no row on the master and slave
[16:20:41] <marostegui>	 inserting a row, but using REPLACE INTO
[16:20:58] <marostegui>	 that gets translated to INSERT into on the binlog, and thus, the slave doesn't break
[16:21:29] <jynus>	 as I said, the translation is a mysqlbinlog invention
[16:22:36] <marostegui>	 yes, but in this case it helps to understand why it is not breaking
[16:24:31] <jynus>	 note all those weirdnesses
[16:24:37] <jynus>	 do not break things further
[16:24:43] <jynus>	 as in, create more drift
[16:24:58] <marostegui>	 yeah, it is actually a good thing
[16:25:00] <jynus>	 but it would be nice to have some log of "I did this"
[16:25:11] <jynus>	 but the expected previous state was not right
[16:30:57] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3893886 (10chasemp) >>! In T184540#3892358, @jcrespo wrote: > The thing is, we already log on SAL whenever we reboot the server- the...
[16:52:26] <wikibugs>	 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3893988 (10jcrespo) I can block all user connections while I run mysql_upgrade- it is a silly thing to do that we do not do anywhere,...
[16:56:01] <jynus>	 db1095 is so outdated that it still uses init.d for managing mariadb
[18:14:11] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1059 BBU issues - https://phabricator.wikimedia.org/T184160#3894345 (10Cmjohnson) Swapped the bbu....leaving this open to confirm everything is okay.
[18:15:29] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1059 BBU issues - https://phabricator.wikimedia.org/T184160#3894351 (10jcrespo) 05Open>03Resolved icinga check says things are ok-  we will reopen if they reappear. Thank you for the help!
[18:28:33] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891821 (10jcrespo) Note creating new tables is not a risky operation and it can be done by anyone with access to the cluster. I will give...