[01:03:39] <wikibugs>	 10DBA, 06Labs, 10Tool-Labs, 13Patch-For-Review: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2730527 (10Dispenser) Queries are being killed after 2 hours since June, leaving data for tools stale.
[06:20:43] <wikibugs>	 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2730655 (10Marostegui) db1069's ALTER is now done so I believe we can close this ticket:  ``` MariaDB MARIADB db1069 commonswiki > show create table revision\G *************************** 1. row ******************...
[06:29:12] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2037: Disk in predictive failure - https://phabricator.wikimedia.org/T148373#2730656 (10Marostegui) And everything looks good now, thanks a lot @Papaul   ``` =>  ctrl slot=0 physicaldrive all show  Smart Array P420i in Slot 0 (Embedded)     array A        physicaldrive 1I:...
[06:29:25] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2037: Disk in predictive failure - https://phabricator.wikimedia.org/T148373#2730657 (10Marostegui) 05Open>03Resolved
[07:31:19] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2730717 (10jcrespo) I suggest we should avoid providing wildcard grants. Those have created issues in the past, and we want to move away from them. When g...
[08:08:20] <wikibugs>	 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2730743 (10Marostegui) The import finished. One table crashed though, however everything looks fine. This is the timeline:  `transcode`table (which as a PK) crashed while importing. Started with innodb_force_re...
[08:09:28] <marostegui>	 jynus ^ I have edited the comment (I hit enter by mistake) just in case you only got to "S4", so just refresh :)
[08:10:33] <jynus>	 I assume every time things crases, no replication was running, or was it?
[08:10:46] <marostegui>	 No, it wasn't running no
[08:11:01] <marostegui>	 configuring and starting replication was the last step :)
[08:11:15] <jynus>	 but maybe running for other shards?
[08:11:41] <marostegui>	 for other shards yes, and I left it running because I wanted to see if we can import data without stopping replication from other sources
[08:11:44] <marostegui>	 Which should be that way
[08:11:49] <marostegui>	 Otherwise it is a bit crappy
[08:11:52] <marostegui>	 (I think)
[08:11:55] <jynus>	 and gtid on or off?
[08:12:05] <marostegui>	 For S3 ON
[08:12:14] <marostegui>	 For S1 I will enable it now 
[08:12:16] <jynus>	 the one that was running?
[08:12:23] <marostegui>	 yep, S3 was the one that was running
[08:12:28] <marostegui>	 S1 was the one being imported
[08:12:32] <jynus>	 then it should be secure
[08:12:46] <jynus>	 gtid in mariadb does provide innodb transactional control
[08:13:25] <jynus>	 transcode, what about this table?
[08:13:39] <jynus>	 no PK like the others, or any other strange characteristic?
[08:13:41] <marostegui>	 Don't know really, first time it crashes during these imports
[08:13:46] <marostegui>	 No no, it does have a PK
[08:14:07] <jynus>	 we should do more tests on eqiad
[08:14:28] <jynus>	 probably there is some kind of ancient row format issue with certain tables
[08:14:38] <marostegui>	 I am going to add to the documentation to the well known issues the PK thingy as it is clearly an issue (or can be)
[08:14:41] <jynus>	 and all of codfw probably comes from the same original server
[08:14:54] <jynus>	 I do not know
[08:14:58] <marostegui>	 Could be, but this table never crashed before while change_tag crashed everytime
[08:15:01] <marostegui>	 it is strange
[08:15:22] <jynus>	 what I am a bit disappointed is with importing tablespaces
[08:15:32] <jynus>	 which doesn't seem so great for us
[08:15:35] <marostegui>	 Yes, it doesn't look super reliable
[08:15:41] <jynus>	 slow, unreliable, yes
[08:15:47] <marostegui>	 Although, this time it worked, we will see what happens with S4
[08:15:57] <marostegui>	 Maybe it was just that problematic table: change_tags
[08:16:04] <jynus>	 it would be nice to see how it works in reverse
[08:16:13] <marostegui>	 in reverse?
[08:16:20] <jynus>	 because if xtrabackup creates problems with many tables
[08:16:47] <jynus>	 these servers as the source of copies seems not a great idea
[08:17:34] <jynus>	 and then we have compression
[08:17:43] <jynus>	 which I do not think will make things more stable...
[08:17:49] <marostegui>	 indeed
[08:18:04] <jynus>	 everything you are doing is great
[08:18:12] <marostegui>	 Thanks for your support :)
[08:18:17] <jynus>	 I am just thinking the best way to move forward
[08:18:28] <marostegui>	 At some point, maybe it is even something we need to consider to create a source host built from a logical backup
[08:18:39] <marostegui>	 I think we can enable compression in S1 now
[08:18:40] <jynus>	 oh, yes
[08:18:51] <jynus>	 we just never find the time
[08:19:03] <marostegui>	 And see how it works with tables that have been copied via moving .ibd files
[08:19:07] <marostegui>	 As I am a bit scared
[08:19:07] <jynus>	 takes 1-2 days
[08:19:22] <marostegui>	 Because if that fails, then that is a blocker
[08:19:38] <marostegui>	 I think it is better to try that before importing S4
[08:19:54] <marostegui>	 So we can have a full picture of how transportable spaces + compression works
[08:19:54] <jynus>	 note we could have been carrying 4.x unsupported formats for a long time
[08:20:21] <jynus>	 or maybe our versions has bugs on features like this, which are not so popular
[08:20:30] <marostegui>	 could be
[08:20:43] <marostegui>	 In that case, the source host needs to be built from logical backups
[08:20:49] <marostegui>	 Which is the safest thing probably
[08:20:54] <jynus>	 also
[08:21:07] <jynus>	 I would love to have per-db tablespaces for s3
[08:21:25] <jynus>	 but that is 5.7-only
[08:21:54] <jynus>	 we are getting better, at least we will not have 40K files on a single dir
[08:21:57] <jynus>	 as with tokudb
[08:22:01] <marostegui>	 yeah XDDD
[08:22:07] <marostegui>	 by the way ,db1069 finished the alter
[08:22:16] <jynus>	 saw it
[08:22:20] <jynus>	 resolve
[08:22:27] <marostegui>	 TokuDB didn't play trick on us this time XD
[08:22:33] <jynus>	 ha
[08:22:39] <jynus>	 was it online?
[08:22:46] <marostegui>	 No XD
[08:22:56] <marostegui>	 But I was expecting it to crash or soemthing as we were manipulating PKs
[08:23:03] <jynus>	 look, when even innodb is more online
[08:23:12] <marostegui>	 My faith on tokudb is quite down now :(
[08:23:12] <jynus>	 than toku, which is one of its selling points
[08:23:46] <jynus>	 it is not that the balance is weighted, it is that there is no reason to use it
[08:23:51] <wikibugs>	 10DBA: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#2730764 (10Marostegui)
[08:23:54] <wikibugs>	 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2730763 (10Marostegui) 05Open>03Resolved
[08:24:17] <jynus>	 for innodb compression
[08:25:25] <jynus>	 there was a task
[08:27:15] <marostegui>	 https://phabricator.wikimedia.org/T139055
[08:27:16] <marostegui>	 this?
[08:27:23] <jynus>	 yes
[08:27:29] <jynus>	 it didn't show on searches
[08:27:40] <marostegui>	 Nop, it didn't I had to go tot he DBA dashboard :|
[08:28:52] <jynus>	 so apparently I tested it on production on db1073
[08:28:56] <jynus>	 before: https://phabricator.wikimedia.org/T139055#2418115
[08:29:14] <jynus>	 ALTER TABLE revision row_format=compressed key_block_size=8\G
[08:29:28] <marostegui>	 16 hours nice
[08:29:45] <jynus>	 (13 hours 10 min 8.65 sec) for pagelinks
[08:29:49] <marostegui>	 I am so non confident about transportable spaces + compression...
[08:29:54] <marostegui>	 I hope MariDB proves me wrong
[08:30:02] <jynus>	 ha
[08:30:39] <marostegui>	 Probably I will stop slave; to alter them
[08:30:48] <jynus>	 db2034 was cloned from db1073
[08:32:24] <jynus>	 https://grafana.wikimedia.org/dashboard/db/server-board?from=1464602836457&to=1468447311389&var-server=db1073&var-network=eth0&panelId=17&fullscreen
[08:32:35] <jynus>	 Note the increase in size before the reduction
[08:33:00] <marostegui>	 Why is that?
[08:33:38] <wikibugs>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#2730768 (10hoo)
[08:33:39] <marostegui>	 is that the table rebuilds during the alters?
[08:36:00] <jynus>	 yes
[08:36:15] <jynus>	 so it is dangerous to do it with low space
[08:36:25] <marostegui>	 yeah
[08:36:39] <marostegui>	 for dbstores we should be safe, but it is indeed something to keep in mind
[08:36:42] <jynus>	 how much space do you have now used/free there?
[08:37:04] <marostegui>	 A lot, 3.2T available but I have around 600G of the snapshot
[08:37:19] <marostegui>	 So 3.8T
[08:37:22] <jynus>	 ok, then used by the fs?
[08:37:32] <jynus>	 3.8 by 3 shards without compression?
[08:37:38] <jynus>	 is that right^?
[08:38:06] <marostegui>	 2 shards (S1, S3) without compression are 2.9T
[08:38:16] <marostegui>	 And the snapshot of those two compressed is 600G
[08:38:24] <jynus>	 ah, ok
[08:38:29] <marostegui>	 root@dbstore2001:/srv# df -hT /srv/
[08:38:29] <marostegui>	 Filesystem            Type  Size  Used Avail Use% Mounted on
[08:38:29] <marostegui>	 /dev/mapper/tank-data xfs   6.6T  3.4T  3.2T  53% /srv
[08:38:49] <marostegui>	 So we should be fine to compress S1
[08:43:01] <wikibugs>	 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2730773 (10jcrespo) db1073 as of now: ``` +----------+------------------+--------+------------+----------+--------+--------+---------+---------+ | DATABASE | TABLE            | ENGINE | ROW_FORMAT | ROWS     | DATA   | IDX    | TOTAL   |...
[08:43:11] <jynus>	 see my update here^
[08:43:35] <jynus>	 it seems I only compressed the top 14 tables
[08:43:43] <jynus>	 to get a 50% reduction
[08:45:22] <jynus>	 clearly, just doing *links tables + revision is a huge help
[08:45:52] <jynus>	 because the amount of text they have
[08:46:17] <jynus>	 so do you have more or less a plan?
[08:46:38] <jynus>	 whatever you learn about compression specifically, add it to that ticket, too
[08:51:38] <marostegui>	 Yeah, I do have a plan
[08:51:50] <marostegui>	 So I am going to wait for S1 to catchup, create a snapshot and start compressing S1
[08:51:54] <marostegui>	 I will add it to the ticket
[08:53:18] <marostegui>	 revision         | InnoDB | Compressed | 679.52M 
[08:53:24] <marostegui>	 Almost 700M rows, amazing
[08:54:15] <jynus>	 uf, mechanical disks + snapshot ?
[08:54:43] <marostegui>	 Creating the tar with pigz last time took around 2 hours or so
[08:54:58] <jynus>	 nothing to object, I just will hope the alter will finish at some point
[08:55:02] <jynus>	 :-)
[08:55:11] <marostegui>	 XDDDD
[08:55:24] <jynus>	 and that you will not run out of snapshot space?
[08:55:42] <marostegui>	 I will delete the previous snapshot
[08:55:55] <marostegui>	 So we would have around 3.6T to start doing the alters (one by one)
[08:56:15] <marostegui>	 I mean, automatically but on a sequential way I think
[08:57:00] <jynus>	 es2015 is not ok :-/
[08:57:19] <marostegui>	 raid again? :(
[08:57:19] <jynus>	 "WARNING: could not read sysctl settings"
[08:57:30] <marostegui>	 :(
[08:57:37] <jynus>	 that should be purely software
[08:57:47] <jynus>	 kernel, or whatever
[08:57:56] <marostegui>	 Interesting sysctl -a does work
[08:58:05] <marostegui>	 maybe it is icinga itself?
[08:58:11] <jynus>	 but it happend last time when there were hw issues
[08:58:19] <marostegui>	 Yeah, it happened to db1082 too
[08:58:22] <jynus>	 maybe I am just being irrational here
[08:58:36] <jynus>	 because that does not make sense
[08:58:39] <marostegui>	 No, that did happen to db1082 when it was having its crisis
[10:01:23] <marostegui>	 jynus: you joining me in the call? :)
[10:01:29] <jynus>	 yes
[10:01:33] <marostegui>	 :)
[10:01:46] <jynus>	 I just see to events
[10:01:49] <jynus>	 2
[10:01:51] <marostegui>	 ah
[10:01:55] <marostegui>	 the DBA triage one
[10:02:00] <marostegui>	 We are both there
[10:02:15] <jynus>	 I think itis the same hangout
[10:02:25] <marostegui>	 should be
[10:09:22] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Wikidata, 07Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010#2730915 (10hoo)
[10:45:22] <wikibugs>	 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2730965 (10mark)
[10:54:42] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2730969 (10mark)
[11:54:29] <marostegui>	 jynus: I lost you
[11:54:38] <jynus>	 let me try to reconnect
[11:54:45] <marostegui>	 ok
[12:48:42] <jynus>	 Progress: 31.151 Time: 10411 / Progress: 15.940 Time: 10411
[12:51:45] <marostegui>	 Not bad
[12:51:53] <jynus>	 https://grafana-admin.wikimedia.org/dashboard/db/server-board?panelId=17&fullscreen&var-server=db1053&var-network=bond0&from=now-7d&to=now-4m
[12:53:46] <marostegui>	 We will see the final graph..
[13:29:26] <wikibugs>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1046 - https://phabricator.wikimedia.org/T148633#2731366 (10elukey) p:05Triage>03High
[13:52:14] <wikibugs>	 10DBA, 10MediaWiki-Database, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2731432 (10Johan)
[14:18:01] <wikibugs>	 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1046 - https://phabricator.wikimedia.org/T148633#2731507 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson The disk was replaced yesterday.  All systems go  root@db1046:~# megacli -PDList -aALL |grep "Firmware state" Firmware state: Online, Spun Up Firmw...
[14:44:20] <jynus>	 do you know what is the deal with s3 on dbstore2001?
[14:45:19] <marostegui>	 jynus: I am updating the ticket right now :_(
[14:45:21] <marostegui>	 But this is the deal
[14:45:24] <marostegui>	 All of a sudden
[14:45:32] <marostegui>	 161020 14:36:56 [ERROR] Master 's3': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 0-171974683-4736235774, which is not in the master's binlog', Internal MariaDB error code: 1236
[14:45:36] <marostegui>	 161020 14:36:56 [Note] Master 's3': Slave I/O thread exiting, read up to log 'db2018-bin.002452', position 4; GTID position 0-171974683-4736235774
[14:45:41] <marostegui>	 just like that
[14:45:47] <marostegui>	 And s3 has been working fine for days
[14:46:12] <jynus>	 what if you disable GTID?
[14:49:38] <marostegui>	 I was first checking the binlogs in the master
[14:49:43] <marostegui>	 to see if that position exists
[14:54:06] <marostegui>	 I guess I can do a change master to and use db2018-bin.002452', position 4;
[14:54:14] <marostegui>	 But I am pretty sure that is wrong and won't work
[14:54:43] <jynus>	 don't
[14:55:01] <jynus>	 first, is id 171974683 the server id of the master?
[14:56:29] <marostegui>	 No, the master's id: 180359174 (db2018)
[14:56:32] <jynus>	 second, try to identify the last executed event of s3 from the binlogs of the dbstore
[14:56:43] <jynus>	 ah, then it may be something else
[14:56:51] <jynus>	 like a local write
[14:57:09] <jynus>	 I think disabling GTID will work
[14:57:26] <jynus>	 but try first to check the above^ to be sure
[14:59:51] <marostegui>	 I am still exploring the master binlog to see if any of them contains 171974683
[15:01:29] <jynus>	 I think it will be on the slave only, by experience
[15:01:48] <marostegui>	 Yeah, it is there indeed
[15:01:50] <marostegui>	 in the salve
[15:01:51] <marostegui>	 slave
[15:02:00] <jynus>	 so that's it
[15:02:09] <marostegui>	 https://phabricator.wikimedia.org/P4275
[15:02:12] <marostegui>	 that is in the slave
[15:12:44] <marostegui>	 It is interesting that that query isn't even on the master's binlogs 
[15:22:17] <marostegui>	 jynus: What do you suggest? Trying to do a change master to master_user_gtid=slave_pos ?
[15:22:37] <jynus>	 to none
[15:22:46] <jynus>	 or whatever is disabling it
[15:23:00] <marostegui>	 Ah, yes: "no"
[15:23:05] <marostegui>	 That is the one to disable it I believe
[15:23:32] <jynus>	 I remember somthing similar happening to a dbstore
[15:23:50] <marostegui>	 all of sudden like this?
[15:24:04] <jynus>	 not all of a sudden
[15:25:22] <marostegui>	 As expected, duplicate entries are now showing up
[15:40:52] <jynus>	 strange
[15:42:13] <marostegui>	 Yeah, it is very strange that it crashed like that
[15:42:26] <jynus>	 there is something writing where it shouldn't
[15:42:44] <jynus>	 we need to know what and where
[15:44:19] <marostegui>	 I am connecting dots now
[15:44:28] <marostegui>	 Because S1 got a duplicate entry as well there, pretty much at the same time
[15:44:43] <marostegui>	 (I am still updating the ticket) but I am giving you a break news :p
[15:57:12] <wikibugs>	 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2731817 (10Marostegui) Looks like after the crash today, S3 replication thread got broken - reminder: s3 has been untouched for almost a week after it was copied over:  ``` 161020 14:23:02 [ERROR] Master 's3':...
[15:58:42] <jynus>	 if it is after the crash, it is not somthing
[15:59:04] <jynus>	 if a server crashes and does not use gtid, it immediately gets corrupted
[15:59:16] <marostegui>	 jynus: s3 was using gtid :)
[15:59:18] <jynus>	 something external
[15:59:44] <jynus>	 then we have a configuration problem
[16:00:18] <marostegui>	 It is weird that after a week replicating just fine, all of a sudden it gets that error
[16:00:26] <jynus>	 mmmm
[16:00:27] <marostegui>	 (there have been many crashes)
[16:00:35] <jynus>	 what is the default database for that change?
[16:00:44] <jynus>	 the one on the ticket?
[16:00:53] <marostegui>	 what do you mean?
[16:01:13] <jynus>	 I am thinking of a filtering issue
[16:01:37] <marostegui>	 filtering as in mysql replication filters?
[16:01:51] <jynus>	 yes
[16:02:03] <marostegui>	 Interesting…let me know what you are thinking :)
[16:02:33] <marostegui>	 This is what S3 has: Replicate_Wild_Do_Table: %wik%.%,heartbeat.%
[16:02:50] <jynus>	 yes, that is why I mentioned
[16:02:51] <jynus>	 it
[16:02:57] <marostegui>	 But you think that issue can happen after a week replicating finely?
[16:03:21] <jynus>	 do you know which server has server id 171974683 ?
[16:03:53] <marostegui>	 No idea, but I can try to check :)
[16:06:24] <marostegui>	 db1057.eqiad.wmnet
[16:06:25] <marostegui>	 server_id      	171974683
[16:06:42] <jynus>	 that is the master
[16:07:45] <marostegui>	 mmm now that I think about it, I checked the master's (codfw) binlog, but not the upstream one
[16:08:00] <marostegui>	 It was not appearing in codfw one (which is weird) but it might appear in the primary one
[16:08:09] <marostegui>	 Which still doesn't make sense to me
[16:08:10] <jynus>	 what is the intermediate master?
[16:08:13] <jynus>	 2016?
[16:08:37] <marostegui>	 for s3 db2018
[16:08:49] <marostegui>	 s3 is the one that had the weird issue
[16:21:53] <jynus>	 too much table corruption
[16:22:05] <jynus>	 I do not even think this is a wise method to continue
[16:23:03] <marostegui>	 I really cannot believe this transportable spaces is that bad :(
[16:23:15] <jynus>	 try 5.7
[16:23:47] <marostegui>	 But we'd need another host to be 5.7 to use as source
[16:24:18] <jynus>	 we will just try with 10 -> 5.7
[16:24:26] <jynus>	 do you have a better suggestion?
[16:24:41] <marostegui>	 No, we can try that
[16:24:48] <jynus>	 because my other suggestion is to do a logical import
[16:24:57] <jynus>	 and you will like that less
[16:24:58] <marostegui>	 Let's try 10.0 -> 5.7
[16:25:00] <marostegui>	 as that can be faster
[16:25:01] <marostegui>	 XDDD
[16:25:04] <jynus>	 :-/
[16:25:17] <marostegui>	 I will go for 5.7 in dbstore2001 then
[16:25:23] <jynus>	 go for a smaller sample
[16:25:30] <marostegui>	 Yeah
[16:25:41] <jynus>	 if a single table doesn't work, it is not worth to continue
[16:25:48] <marostegui>	 indeed
[16:26:02] <marostegui>	 Well, for S1 for instance, it failed 1 table out of a lot
[16:26:04] <jynus>	 what script are you using anyway?
[16:26:35] <marostegui>	 For moving the tablespaces?
[16:28:11] <jynus>	 for whatever you are doing
[16:28:39] <marostegui>	 Ah, for the normal copies (not moving ibd) just a single nc and for moving the ibd files I am doing it manually too, as I do not want to add more variables
[16:28:43] <marostegui>	 ie: the script might be broken
[16:28:48] <jynus>	 no
[16:28:53] <marostegui>	 So I am executing the MySQL commands with a bash oneliner
[16:28:57] <jynus>	 but how are you automatizing
[16:29:00] <jynus>	 each table
[16:29:05] <marostegui>	 Ah
[16:29:09] <marostegui>	 bash so far
[16:29:11] <jynus>	 I suppose you do not run 40000 commands?
[16:29:16] <marostegui>	 haha no XDDD
[16:29:22] <jynus>	 is it on the documentation?
[16:29:23] <marostegui>	 Just simple bash oneliners
[16:29:26] <marostegui>	 I think so
[16:29:39] <marostegui>	 yep: 	•	for i in `mysql --skip-ssl enwiki -e "show tables;" -B`; do echo $i; mysql --skip-ssl enwiki -e "flush table $i for export;";done
[16:31:56] <marostegui>	 I am going to do some groceries - will talk to you tomorrow!!
[16:31:58] <jynus>	 wait
[16:32:01] <marostegui>	 what
[16:32:16] <jynus>	 I think there are things there that should not work
[16:32:28] <marostegui>	 like?
[16:33:09] <jynus>	 go, I will have a closer look at it
[16:33:18] <marostegui>	 sure!
[16:33:26] <marostegui>	 let me know if you see something strange
[16:33:30] <marostegui>	 thanks!
[16:37:41] <jynus>	 if you are doing as documented, I would not know how it can even work?
[16:38:09] <jynus>	 flush table $i for export; should be done within a session
[16:38:24] <jynus>	 if you exist mysql, the locks should be dropped
[16:38:34] <jynus>	 and mysql cannot be stopped
[16:38:49] <jynus>	 plus you are coping ibdata1 over the other?
[16:39:09] <jynus>	 it will crash, and when it restarts, it will not likely work again
[17:03:05] <jynus>	 I think the last part you aren't, it is just a typo of the documentation
[17:03:20] <jynus>	 but let's talk tomorrow about improving the method :-)
[17:04:42] <jynus>	 check before I connect https://www.percona.com/blog/2014/12/09/mysql-5-6-transportable-tablespaces-best-practices/
[17:13:57] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2732127 (10AlexMonk-WMF) >>! In T148560#2730717, @jcrespo wrote: > Right now we maintain a blacklist on realm.pp. We should transform that into a white li...
[17:21:42] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2732172 (10AlexMonk-WMF) I suppose we go from one list that needs updating when private wikis are created to one that needs updating when non-private (the...
[17:23:19] <wikibugs>	 10DBA, 06Community-Tech, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in production - https://phabricator.wikimedia.org/T148243#2718099 (10kaldari)
[18:12:22] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: es2015 crashed with no os logs (kernel logs or other software ones) - it shuddenly went down - https://phabricator.wikimedia.org/T147769#2732383 (10Papaul) p:05High>03Normal
[20:04:53] <marostegui>	 jynus: Yeah - I was not copying the ibdata1 :-)
[20:05:05] <marostegui>	 But what you said about closing the session can be the key. However, S1 is working fine :|
[20:05:14] <marostegui>	 Let's discuss that tomorrow yeah
[20:05:19] <marostegui>	 Thanks for the feedback though!
[20:20:52] <wikibugs>	 10DBA: apply events killing long running queries to db1070; any other production server - https://phabricator.wikimedia.org/T148790#2732802 (10jcrespo)
[20:21:20] <wikibugs>	 10DBA: apply events killing long running queries to db1070; any other production server - https://phabricator.wikimedia.org/T148790#2732815 (10jcrespo) p:05Triage>03Unbreak!
[22:04:50] <wikibugs>	 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2733044 (10Deskana)
[22:05:41] <wikibugs>	 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2710120 (10Deskana) The above is a hack that unblocks @smalyshev's work temporarily, but we are awaiting a p...