[05:23:43] <wikibugs>	 10DBA, 10Operations: Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it - https://phabricator.wikimedia.org/T57385 (10Marostegui) 05Open→03Resolved >>! In T57385#5310801, @ArielGlenn wrote: > Um, it has? I just found it on meta, though empty. >  >   wikiadmin@1...
[05:23:49] <wikibugs>	 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui)
[05:34:38] <wikibugs>	 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) After this big batch of wiki compression only 3555 tables were left to be compressed -  I am now trying to compress medium size wikis, between 20G and 100GB (a total of 3000 tables)...
[05:48:43] <wikibugs>	 10DBA, 10Patch-For-Review: Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required) - https://phabricator.wikimedia.org/T227062 (10Marostegui) I have restarted db1109 to pickup STATEMENT as a binlog format. db1109 will be the candidate master once db1104 (current candidate master) get...
[08:03:28] <jynus>	 performance_schema | OFF on all pc* hosts, I just realixed
[08:03:32] <jynus>	 *realized
[08:05:57] <marostegui>	 I wonder if it was ever enabled on the old ones
[08:15:59] <jynus>	 I am not touching that for the moment, but I am installing sys on all hosts
[08:16:07] <jynus>	 for consistency
[08:16:10] <marostegui>	 sounds good
[08:16:24] <jynus>	 even if sys.processlist will return 0 rows with it disabled
[08:16:39] <jynus>	 because now there are hosts on the same section, some with it installed
[08:16:42] <jynus>	 and some with it not
[08:17:18] <jynus>	 or even instances on the same hosts
[08:23:31] <jynus>	 db1065, version: 10.1.33, up: 1y, RO: OFF, binlog: MIXED, lag: None, processes: None, latency: 0.0993
[08:24:24] <marostegui>	 \o/
[08:24:29] <marostegui>	 what was the issue with the snapshots btw?
[08:24:31] <marostegui>	 I didn't check
[08:24:36] <marostegui>	 was it a metadata issue?
[08:25:06] <jynus>	 I think backups get stuck on db2097
[08:25:10] <jynus>	 the host with memory issues
[08:25:17] <marostegui>	 Ah right
[08:25:28] <marostegui>	 The DIMM is supposed to arrive today as per the last message from papaul
[08:25:33] <jynus>	 and that creates an exception
[08:26:12] <jynus>	 maybe I should move s3 backups higher
[08:26:14] <jynus>	 as it takes more time
[08:26:21] <jynus>	 higher == earlier
[08:26:41] <marostegui>	 so it doesn't "collide?"
[08:27:10] <jynus>	 also because the order is not perfect, and it may end up executing that on a single thread
[13:28:11] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/521266/
[13:36:53] <marostegui>	 and for the x1 codfw failover: https://gerrit.wikimedia.org/r/#/c/521273/
[14:18:33] <marostegui>	 let's do x1?
[14:18:39] <marostegui>	 when you are around
[14:19:03] <jynus>	 one sec
[14:21:29] <marostegui>	 sure!
[14:36:03] <jynus>	 see my amend, it is the best way to explain :-)
[14:36:36] <marostegui>	 Aaaaah!!!
[14:36:38] <marostegui>	 yes!
[14:36:41] <marostegui>	 I see :)
[14:36:53] <jynus>	 there was still some mistakes on the config
[14:36:54] <marostegui>	 thanks :)
[14:36:58] <jynus>	 I haven't checked all
[14:37:15] <jynus>	 I've also left m4 with it disabled
[14:37:20] <marostegui>	 yeah
[14:37:21] <jynus>	 as it was explicitly like that
[14:37:26] <marostegui>	 it will go away anyways
[14:37:43] <jynus>	 see now if you get what I meant
[14:38:00] <marostegui>	 Yeah
[14:38:02] <jynus>	 sometimes is easier to code than to try to explain
[14:38:04] <marostegui>	 Now I do :)
[14:38:07] <marostegui>	 definitely 
[14:38:11] <jynus>	 the template has an if--then--else
[14:38:16] <jynus>	 on purpose
[14:38:20] <marostegui>	 I am running a puppet compiler
[14:38:23] <jynus>	 that enables user_stats
[14:38:28] <jynus>	 when dit disables p_s
[14:38:33] <jynus>	 *it
[14:38:35] <marostegui>	 yeah, I wasn't sure if that was actually intended or what
[14:39:02] <jynus>	 the # we are testing impact of performance schema
[14:39:07] <jynus>	 wasn't intended
[14:39:39] <marostegui>	 I am going to merge, puppet compiler looks good
[14:39:43] <marostegui>	 ok?
[14:40:24] <jynus>	 up to you, it just needs your review
[14:40:32] <marostegui>	 yeah, I just checked :)
[14:40:33] <jynus>	 and it should be a safe change
[14:41:04] <marostegui>	 Going to apply it, run puppet and restart db1132 and then we can move to x1
[14:43:21] <jynus>	 when you finish, I thought a bit about https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/521226
[14:43:41] <jynus>	 and I think we should apply it (although doesn't have to be today)
[14:43:49] <jynus>	 because this is a switchover script
[14:43:59] <jynus>	 not a failover one (on my way of speaking)
[14:44:19] <jynus>	 when there is a faiover one, flexibility should be there indeed
[14:44:34] <jynus>	 but not for a "best practices" scheduled script
[14:47:26] <marostegui>	 Yeah, I know
[14:47:41] <marostegui>	 I guess the cases where we'll go for a master with a different binlog format are kinda reduced
[14:49:20] <jynus>	 an option could be added
[14:49:29] <jynus>	 but I hate too many options
[14:49:33] <marostegui>	 yeah, I think we need the --force at some point
[14:49:37] <marostegui>	 but probably not needed now
[14:49:44] <marostegui>	 let's play with x1 codfw then? :)
[14:49:47] <jynus>	 a single forve I would be ok with
[14:49:54] <jynus>	 also maybe a --test-only
[14:50:04] <marostegui>	 yeah maybe
[14:50:07] <marostegui>	 that's a good one
[14:51:22] <marostegui>	 so, db2069 will be the new codfw master for x1
[14:51:29] <marostegui>	 so I will tell you what I want to do 
[14:51:33] <marostegui>	 and you tell me if it is correct
[14:51:38] <wikibugs>	 10DBA, 10MediaWiki-General-or-Unknown, 10Core Platform Team Kanban (Clinic Duty Team): Investigate query planning in MariaDB 10 - https://phabricator.wikimedia.org/T85000 (10Anomie) 05Open→03Declined > which may kill performance for a lot of API queries. We should figure out if that's true  I guess it wa...
[14:52:12] <marostegui>	 I will do it in the same way I will be doing it tomorrow with m2 in eqiad
[14:52:38] <marostegui>	 so first: move_replica.py db2115 db2069
[14:52:54] <marostegui>	 and that with all the codfw slaves to put them under db2069
[14:53:57] <marostegui>	 once that is done and all the slaves are under db2069 I plan to do: switchover.py --replicating-master --read-only-master db2045 db2069
[14:54:04] <marostegui>	 db2045 is the current master
[14:54:30] <jynus>	 please do, downtime if you need something
[14:54:37] <marostegui>	 yeah of course
[14:54:39] <jynus>	 and let's check coords and compare later
[14:54:47] <marostegui>	 but the above sounds good, right?
[14:55:16] <jynus>	 yes
[14:55:19] <marostegui>	 cool
[14:55:27] <marostegui>	 I will let you know how it goes :)
[14:55:28] <jynus>	 wait
[14:55:32] <marostegui>	 waiting
[14:55:37] <jynus>	 if it is in 2 steps
[14:55:44] <jynus>	 it is --only-slave-move
[14:55:48] <jynus>	 and later --skip-slave-move
[14:56:18] <marostegui>	 --only-slave-move? 
[14:56:22] <marostegui>	 move_replica doesn't have that
[14:56:32] <marostegui>	 or you want to use switchover for both steps?
[14:56:36] <jynus>	 only-slave move is for switchover
[14:56:44] <jynus>	 it basically calls move()
[14:56:48] <marostegui>	 Ah ok
[14:56:59] <jynus>	 just handles the what on its own
[14:57:12] <marostegui>	 so I can do switchover.py --only-slave-move db2115 db2069
[14:57:15] <jynus>	 they call essentially the same method in the end
[14:57:19] <marostegui>	 cool
[14:57:54] <marostegui>	 I will add these commands to the etherpad checklist for tomorrow too
[14:58:00] <jynus>	 but switchover does all checks
[14:58:09] <marostegui>	 great - I will use that then
[14:58:10] <jynus>	 just to be sure
[14:58:28] <jynus>	 but it wouldn't really be a huge difference
[14:58:42] <jynus>	 although I think using switchover may be easier
[14:58:45] <marostegui>	 totally
[14:58:48] <marostegui>	 better to use just one
[15:00:14] <jynus>	 although --only-slave-move doesn't ask for confirmnation
[15:01:03] <marostegui>	 I think I am not understanding correctly the essence of this
[15:01:10] <marostegui>	 Let's see if you can help me!
[15:01:19] <marostegui>	 So, I want to move db2115 under db2069
[15:01:34] <marostegui>	 As per the usage, I understand I need to use: switchover.py --replicating-master --read-only-master --only-slave-move db2069.codfw.wmnet db2115.codfw.wmnet
[15:01:52] <jynus>	 no, switchover always takes the current master and replica
[15:02:01] <jynus>	 and autoamtically searches the current master replicas
[15:02:21] <jynus>	 it is like step 1/2 and step 2/2
[15:03:06] <marostegui>	 So how do I combine switchover.py with a topology change?
[15:03:18] <marostegui>	 Ah, I think I get it
[15:03:24] <jynus>	 db2045 db2115
[15:03:27] <jynus>	 I think it is
[15:03:37] <marostegui>	 No, I guess it is db2045 db2069 
[15:03:40] <marostegui>	 db2069 is the new master
[15:03:43] <marostegui>	 and db2045 is the current
[15:03:44] <jynus>	 sorry
[15:03:56] <jynus>	 then that, and it will move automatically the other 2
[15:04:00] <marostegui>	 And it will move everything under db2069
[15:04:02] <marostegui>	 yeah
[15:04:02] <marostegui>	 right
[15:04:04] <marostegui>	 lovely!
[15:04:11] <jynus>	 it is "easier"
[15:04:22] <jynus>	 but maybe not obvious
[15:04:28] <marostegui>	 it is a lot easier
[15:04:34] <marostegui>	 I am just used to repl.pl :)
[15:04:48] <jynus>	 with move you would have to do every one individually
[15:04:56] <marostegui>	 yeah indeed
[15:04:58] <marostegui>	 so, here we go!
[15:05:01] <jynus>	 and it doesn't do the smisync and other stuff
[15:05:38] <jynus>	 (and that is why the strict checks are handy :-)
[15:06:55] <marostegui>	 SUCCESS: All slaves moved correctly, but not continuing further because --only-slave-move
[15:06:56] <jynus>	 I was running ./replication_tree.py db2045
[15:07:05] <marostegui>	 topology looks good
[15:07:05] <jynus>	 and got the whole process printed
[15:07:08] <marostegui>	 going to do some checks
[15:08:24] <marostegui>	 it looks good :)
[15:09:26] <marostegui>	 ok, so going for: switchover.py --skip-slave-move db2045 db2069 then 
[15:09:36] <jynus>	 look:
[15:09:57] <jynus>	 https://phabricator.wikimedia.org/P8723
[15:10:07] <jynus>	 probably it would be nice to print that on every step :-)
[15:10:24] <marostegui>	 hahaha nice progress bar
[15:10:39] <jynus>	 so the move is the actual dangerous new thing
[15:10:51] <jynus>	 the other steps have been virtually untouched
[15:11:00] <marostegui>	 and it worked nicely!
[15:11:03] <marostegui>	 well done!
[15:11:07] <jynus>	 default timeout?
[15:11:10] <marostegui>	 yeah
[15:11:13] <marostegui>	 I didn't change it
[15:11:22] <jynus>	 reducing it and the parallel stuff should get better times
[15:11:38] <marostegui>	 yeah, but I wanted to see all the checks slowly as we had no rush really
[15:11:41] <jynus>	 +1
[15:11:49] <jynus>	 just as a FYI
[15:12:04] <marostegui>	 doing the failover itself now
[15:12:35] <jynus>	 switchover :-D
[15:12:40] <marostegui>	 :p
[15:12:51] <marostegui>	 ok, done, got an exception
[15:12:56] <jynus>	 oh
[15:13:00] <marostegui>	 but looks like it worked, let me check
[15:13:24] <marostegui>	 at the very end I got: https://phabricator.wikimedia.org/P8724
[15:13:54] <jynus>	 interesting
[15:13:59] <jynus>	 is replication setup?
[15:14:03] <marostegui>	 yeah
[15:14:05] <marostegui>	 I ran this: /switchover.py --replicating-master --read-only-master --skip-slave-move db2045.codfw.wmnet db2069.codfw.wmnet
[15:14:20] <marostegui>	 replication was setup correctly from what I can see
[15:14:21] <jynus>	 on both old and new replica?
[15:14:28] <jynus>	 *master
[15:14:38] <marostegui>	 yeap
[15:14:42] <marostegui>	 no GTID enabled though
[15:14:49] <marostegui>	 on the old master
[15:15:02] <jynus>	 yeah, that happens later
[15:15:12] <marostegui>	 yeah
[15:15:19] <marostegui>	 Same for zarcillo and all that happens later too
[15:16:25] <jynus>	 will look at it when I finish the stop of the host
[15:16:32] <marostegui>	 no worries
[15:16:42] <marostegui>	 I will update zarcillo and tendril and change the events
[15:18:24] <jynus>	 db2101:3320, version: 10.1.39, up: 31d, RO: ON, binlog: STATEMENT
[15:18:34] <jynus>	 although that is not new
[15:18:52] <jynus>	 (the others are ROW)
[15:19:16] <marostegui>	 probably a left over from all the servers setup
[15:19:21] <marostegui>	 I will get that fixed
[15:19:24] <marostegui>	 thanks for reporting it
[15:20:03] <jynus>	 so thinkit it could be something that is expected to be set
[15:20:15] <jynus>	 when not run with --skip
[15:22:52] <marostegui>	 so, for tomorrow, how do you feel we should operate?
[15:23:02] <marostegui>	 I think we can still use the new switchover
[15:23:57] <jynus>	 could you paste the line 271 of the version you used?
[15:24:39] <marostegui>	 sure
[15:24:51] <marostegui>	         if not result['success']:
[15:24:51] <marostegui>	             print('[ERROR]: Original GTID mode was not recovered on the new master')
[15:24:51] <marostegui>	     return 0
[15:25:11] <jynus>	 oh, I see
[15:25:14] <jynus>	 silly mistake
[15:25:20] <jynus>	 that function returns a bool
[15:25:25] <jynus>	 not a result set
[15:25:28] <jynus>	 easy fix
[15:26:47] <jynus>	 sadly, I had not tested that combination
[15:28:38] <jynus>	 because normally a real master does not have gtid enabled
[15:29:08] <marostegui>	 indeed
[15:29:09] <jynus>	 you were using 58efc9201945e63b5ea21a2, right?
[15:29:22] <jynus>	 so to rebase the path on top of that
[15:29:59] <marostegui>	 commit 58efc9201945e63b5ea21a2fab3d0be72ca93771
[15:30:02] <marostegui>	 so yeah
[15:32:52] <jynus>	 https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/521300/1/wmfmariadbpy/switchover.py
[15:32:55] <jynus>	 ^fixed
[15:33:07] <jynus>	 we can try it twice again :-D
[15:33:22] <marostegui>	 haha nice fix!
[15:35:22] <jynus>	 funnily, I had been biten by this on move_replica.py
[15:39:11] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08 - https://phabricator.wikimedia.org/T225378 (10Papaul) 05Open→03Resolved Before  {F29709098} After {F29709100}  This is complete   return tracking information  {F29709113}
[15:39:15] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10Papaul)
[15:39:36] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08 - https://phabricator.wikimedia.org/T225378 (10jcrespo) 05Resolved→03Open a:05Papaul→03jcrespo ` Mem:         515690 ` HW seems to be fixed, owning for the followup (software) st...
[15:39:40] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo)
[15:40:14] <wikibugs>	 10DBA, 10Operations: db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08 - https://phabricator.wikimedia.org/T225378 (10jcrespo)
[15:41:00] <jynus>	 marostegui: if you update to cff87cd we should be good to go
[15:41:16] <marostegui>	 doing it now
[15:41:45] <marostegui>	 good
[15:41:47] <marostegui>	 done
[15:42:32] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) @Marostegui  Thanks
[15:44:11] <jynus>	 want to do another try?
[15:44:23] <marostegui>	 sure
[15:44:33] <marostegui>	 I will do the same 2 steps
[15:44:38] <jynus>	 ok
[15:45:47] <marostegui>	 moving slaves now
[15:47:29] <marostegui>	 slaves moved
[15:47:31] <marostegui>	 doing some checks
[15:47:54] <jynus>	 at least the new thing seems stable
[15:48:09] <jynus>	 handling gtid was also a new thing
[15:48:29] <marostegui>	 ok, let's go for the switchover
[15:48:35] <marostegui>	 the topology looks good
[15:48:57] <marostegui>	 worked like a charm!
[15:49:21] <jynus>	 did it update zarcillo and tendril?
[15:49:26] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10ArielGlenn) We don't really have daily stats of any sort in grafana but maybe we should (or have a place for it to live in any case...
[15:49:31] <marostegui>	 It said it did, I am manually checking
[15:49:38] <jynus>	 he he
[15:49:44] <jynus>	 do not trust the code
[15:49:53] <marostegui>	 haha just checking! it is the first time we use it!
[15:49:59] <jynus>	 indeed
[15:50:25] <marostegui>	 worked fine!
[15:50:26] <marostegui>	 :)
[15:50:30] <marostegui>	 congratulations :*
[15:50:33] <jynus>	 :-D
[15:50:57] <jynus>	 we should leave however compare.py running
[15:51:14] <jynus>	 and saving the binlogs (althoug it will also be on the logs)
[15:51:36] <jynus>	 although with read only mode it is "easy"
[15:52:59] <jynus>	 can I edit the etherpad then?
[15:53:06] <marostegui>	 yeah
[15:53:11] <marostegui>	 I have moved stuff to manual steps
[15:53:18] <marostegui>	 but please, take a look and check it
[15:53:26] <marostegui>	 and same for the patch
[15:53:45] <jynus>	 I have removed the disable gtid
[15:53:50] <jynus>	 as it is done automatically
[15:53:50] <marostegui>	 great
[15:55:37] <jynus>	 So I can see how things like row format and version checks could bee annoying
[15:55:54] <jynus>	 but then it is the question of if we could trust a --force
[15:56:28] <jynus>	 let's leave it aside for now
[15:56:31] <marostegui>	 yeah
[15:56:39] <marostegui>	 I think it is useful on some situations
[15:56:40] <jynus>	 and we should find a way to make things both safe and flexible
[15:56:42] <marostegui>	 at least to have it available
[15:56:55] <jynus>	 yes, but on the other side, not checking could lead to human error
[15:57:03] <jynus>	 not an easy compromise
[15:57:13] <marostegui>	 yeah, by the default it should check and complain
[15:57:20] <marostegui>	 but something like --i-know-what-i-am-doing!
[15:57:39] <jynus>	 but if we use force all the time, checks are wothless
[15:57:53] <marostegui>	 no, it will be the default
[15:58:17] <jynus>	 I don't know, I personally need to think more abou tthe best way t to proceed
[15:58:34] <marostegui>	 yeah
[15:58:38] <marostegui>	 we can continue discussing
[16:22:42] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore) So we (WMDE) actually made a daily namespace in graphite for tracking data daily instead of minutely. We have been using...
[17:04:43] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10ArielGlenn) Are there visible graphs for these? Looking a bit at the repo scripts now. I guess we'd want something in src/commons/c...