[05:07:42] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[05:31:26] <_joe_>	 https://phabricator.wikimedia.org/T222224 should be interesting to you
[05:31:46] <marostegui>	 yep
[05:31:48] <marostegui>	 we are aware of it
[05:31:51] <marostegui>	 thanks :)
[05:39:05] <_joe_>	 we're discussing it a bit at techcom
[05:49:35] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) @aaron @Krinkle @jcrespo I would appreciate a review of ^ - it should be a quick one...
[06:19:40] <wikibugs>	 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) @Cmjohnson I'd need a heads up of ~15 mins before the maintenance to shutdown the host properly, but we can do it anytime!
[06:56:25] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10jcrespo)
[06:59:05] <marostegui>	 will you add db2116 to mwconfig files or you want me to do that?
[07:05:51] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10Marostegui) Nice! I guess I have had too many nightmares with GTID, I need to start trusting it again, this will definitely help! :) Probably this needs to go...
[07:18:45] <jynus>	 the s1 snapshot on codfw worked, the one on eqiad didn't
[07:20:26] <jynus>	 the ones on codfw took 2 hours for s1
[07:24:07] <jynus>	 I am about to finish compressing db2098
[07:24:52] <jynus>	 are you using db1139? Can I downtime it?
[07:25:39] <marostegui>	 I am not using it
[07:25:40] <jynus>	 (it could be the snapshotting script)
[07:26:34] <marostegui>	 I am using db2098:s2 but it will be released in a bit
[07:26:54] <jynus>	 I think the copy failed there and it left the replica stopped
[07:27:05] <marostegui>	 db1139 you mean?
[07:27:14] <jynus>	 yes
[07:28:09] <marostegui>	 do you want me to add db2117 to the mwconfig files?
[07:28:18] <marostegui>	 I need to add a few others
[07:28:19] <jynus>	 I was going to ask you
[07:28:21] <marostegui>	 so I can include it
[07:28:28] <jynus>	 you are leading this
[07:28:40] <jynus>	 I am not going to do anything without your ok
[07:28:58] <marostegui>	 I will add it, I am preparing a patch for other hosts, so I will include db2117
[07:29:29] <jynus>	 I have not done a check of data/grants/etc
[07:31:15] <marostegui>	 I can do that
[07:35:04] <jynus>	 I was wrong, all snapshots failed today
[07:35:11] <jynus>	 on transfer
[07:35:17] <jynus>	 so probably a software bug
[07:37:16] <marostegui>	 db2098:3312 released, feel free to start replication once you are done with compression
[07:48:42] <jynus>	 I am only doing compression on s3
[07:49:07] <jynus>	 I am going to do a test backup of db1139:s1 stopping replication
[07:49:13] <marostegui>	 heh, I just saw a funny thing
[07:49:26] <marostegui>	 You remember the parsercache key names, the IPs
[07:49:39] <jynus>	 ?
[07:49:45] <marostegui>	 '10.64.32.72'  => '10.64.16.20',  # pc1008, B8 4.4TB 256GB # pc2
[07:49:47] <marostegui>	 and
[07:49:52] <marostegui>	 '10.64.48.128' => '10.64.32.29',  # pc1009, C3 4.4TB 256GB # pc3
[07:50:00] <marostegui>	 72.32.64.10.in-addr.arpa domain name pointer db1133.eqiad.wmnet.
[07:50:09] <marostegui>	 128.48.64.10.in-addr.arpa domain name pointer restbase1025-c.eqiad.wmnet.
[07:50:13] <marostegui>	 now those are real IPs
[07:51:45] <jynus>	 https://grafana.wikimedia.org/d/000000274/prometheus-machine-stats?orgId=1&from=1556783494104&to=1557388234105&var-server=db2098&var-datasource=codfw%20prometheus%2Fops
[07:52:13] <jynus>	 better: https://grafana.wikimedia.org/d/000000274/prometheus-machine-stats?orgId=1&from=1556783494104&to=1557388234105&var-server=db2098&var-datasource=codfw%20prometheus%2Fops&panelId=12&fullscreen
[07:52:28] <marostegui>	 oh wow
[07:54:47] <marostegui>	 A sanity check on https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/508992/ would be appreciated :)
[07:57:17] <jynus>	 It had the same problem than you /opt/wmf-mariadb101/bin/mariabackup: Error writing file 'UNKNOWN'
[07:57:30] <jynus>	 there may be something else happening there
[07:57:33] <marostegui>	 interesting
[07:57:39] <marostegui>	 also myisam?
[07:58:36] <jynus>	 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 34: invalid start byte
[07:58:51] <marostegui>	 I didn't get that
[08:03:07] <marostegui>	 thanks :)
[08:09:58] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[08:15:38] <jynus>	 so I have broken the transfer, fixing it
[08:16:02] <marostegui>	 :)
[08:18:38] <jynus>	 a 2 character fix, now it works
[08:18:49] <marostegui>	 haha
[08:19:05] <marostegui>	 that was the cause of the error about utf-8?
[08:19:08] <jynus>	 https://jynus.com/gif/facepalm.gifv
[08:19:14] <jynus>	 I will tell you
[08:19:17] <marostegui>	 A new one!!!!!!!!!!
[08:19:20] <jynus>	 want to shouw you first
[08:19:44] <marostegui>	 we've been too long without a new release of gifv!
[08:20:41] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[08:24:43] <jynus>	 marostegui: spot the difference: https://gerrit.wikimedia.org/r/#/c/operations/software/wmfmariadbpy/+/509001/1/wmfmariadbpy/transfer.py
[08:25:14] <marostegui>	 hahahaha
[08:25:23] <marostegui>	 was it easy to spot while debugging?
[08:25:53] <jynus>	 yes
[08:26:06] <jynus>	 the problem here is the lack of unit testing to prevent regression
[08:28:44] <jynus>	 I can create a tarball with a backup of something you may need
[08:28:59] <jynus>	 any section I can help with?
[08:29:02] <marostegui>	 yes!
[08:29:03] <marostegui>	 s3 :)
[08:29:24] <marostegui>	 (in codfw)
[08:29:24] <jynus>	 on both dcs?
[08:29:43] <marostegui>	 only codfw
[08:29:51] <marostegui>	 it will go to db2105 and db2109
[08:30:23] <jynus>	 ok, let me create one
[08:40:11] <jynus>	 Running XtraBackup at db2098.codfw.wmnet:3313 and sending it to dbprov2002.codfw.wmnet
[08:40:50] <marostegui>	 \o/
[08:42:17] <jynus>	 when it is done on dbprov2002:/srv/backups/snapshot/latest
[08:42:35] <jynus>	 you will be able to use transfer.py --type=decompress
[08:42:47] <marostegui>	 great
[08:43:00] <jynus>	 and it should be much faster as it will be precompressed and pre-prepared
[08:43:51] <marostegui>	 transfer.py --no-checksum --no-encrypt --type=decompress dbprov2002.codfw.wmnet:/srv/backups/snapshots/latest/xxxx.tar.gz db2105.codfw.wmnet:/srv/sqldata ?
[08:43:53] <jynus>	 transfer.py should work back again
[08:44:21] <marostegui>	 Going to add that to the cheatsheet
[08:44:37] <jynus>	 something along those lines, not sure about the destination hosts
[08:44:40] <jynus>	 if you are going to do 2
[08:44:45] <jynus>	 you can do the 2 at the same time
[08:45:19] <jynus>	 .tar.gz db2105.codfw.wmnet:/srv/sqldata db21XX.codfw.wmnet:/srv/sqldata
[08:46:32] <jynus>	 spoiler-- it won't do the 2 at the same time, for the dbprov it will be faster to do both in parallel to test the 10G
[08:46:49] <marostegui>	 what?
[08:46:53] <marostegui>	 it will or it will not?
[08:46:59] <marostegui>	 Added that line to the cheatsheet
[08:47:03] <jynus>	 it intends to be in parallel
[08:47:04] <marostegui>	 Feel free to add/modify more stuff
[08:47:10] <jynus>	 but it is implemented in series
[08:47:42] <marostegui>	 ah I see
[08:47:48] <jynus>	 because it is supposed to do multi-cast or torrent
[08:48:03] <jynus>	 in this case it is better to run it twice at the same time
[08:48:19] <jynus>	 and see if we can use the 10G better
[08:48:31] <marostegui>	 good, let me know when I can run it :)
[08:48:52] <jynus>	 it is on cywikibooks
[08:48:56] <jynus>	 some more time to go
[08:49:10] <marostegui>	 no worries :)
[08:57:29] <wikibugs>	 10DBA, 10Goal: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo)
[08:57:34] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize eqiad and codfw source backup hosts & codfw backup test host - https://phabricator.wikimedia.org/T220572 (10jcrespo) 05Open→03Resolved Compression has finished for these hosts.
[09:17:38] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1133.eqiad.wmnet'] ` The log can be found in `/v...
[09:19:43] <jynus>	 I am going to stop the mysqls on old dbstores
[09:19:51] <marostegui>	 cool
[09:20:13] <jynus>	 this doesn't have to happen now, but you should be thinking if there is something you need to keep from there
[09:20:27] <marostegui>	 yeah, I just pinged the chase and john
[09:20:31] <marostegui>	 to see if they can come back to me
[09:21:02] <jynus>	 I won't send it to dcops until a few weeks pass
[09:21:13] <jynus>	 at the very least
[10:22:50] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1133.eqiad.wmnet'] `  Of which those **FAILED**: ` ['db1133.eqiad.wmnet'] `
[10:35:04] <jynus>	 I'm stupid
[10:36:07] <marostegui>	 your words!
[10:39:18] <jynus>	 https://gerrit.wikimedia.org/r/509034
[10:40:10] <marostegui>	 haha
[10:42:29] <jynus>	 that is some weird diff output: https://gerrit.wikimedia.org/r/#/c/operations/software/wmfmariadbpy/+/509034/2/wmfmariadbpy/transfer.py
[10:51:43] <jynus>	 prepare is running now
[10:51:50] <marostegui>	 \o/
[13:16:42] <marostegui>	 hows the prepare going?
[13:28:19] <jynus>	 Backup s3 generated correctly.
[13:28:38] <jynus>	 thats dvprob2002
[13:28:47] <marostegui>	 nice
[13:28:54] <marostegui>	 So I can trigger the decompression?
[13:29:04] <jynus>	 yes
[13:29:08] <jynus>	 if
[13:29:09] <marostegui>	 \o/
[13:29:16] <jynus>	 code is updated to the last version
[13:29:24] <marostegui>	 ok, will update my git
[13:29:57] <jynus>	 I mean puppet on hosts
[13:30:02] <jynus>	 because before it failed
[13:30:09] <marostegui>	 ah
[13:30:12] <marostegui>	 I will run it then
[13:32:35] <marostegui>	 here we go!
[13:34:44] <jynus>	 is it working?
[13:34:50] <marostegui>	 yeah!
[13:35:12] <marostegui>	 I put the two hosts to do it in "paralell" as you suggested
[13:35:16] <marostegui>	 so so far db2105 is being copied
[13:35:35] <jynus>	 we could change the parallism to be real
[13:35:59] <jynus>	 I didn't thought at the time of the 10G -> 1G use case
[13:47:17] <jynus>	 are you using db2098?
[13:47:24] <marostegui>	 nope
[13:47:34] <jynus>	 I want to restart replication there
[13:47:42] <jynus>	 because of the previous bug
[13:47:45] <marostegui>	 go for it!
[13:49:38] <jynus>	 before you leave, let's please sync on what things are ongoing/pending
[13:50:05] <marostegui>	 Sure, I will work a couple of more hours probably
[13:50:16] <marostegui>	 I will let you know before I leave
[13:50:31] <jynus>	 yeah, I don't mind if it is 5 minutes or 4 hours, I created a bit or a problem with that bug
[13:50:48] <jynus>	 and want to make sure all alerts are accounted for
[13:51:14] <jynus>	 to see if I can leave https://gerrit.wikimedia.org/r/509012 deployed
[13:51:21] <marostegui>	 so from my side I am touching: db1138 (not in tendril yet), db1081, db2105 and db2109 (neither of those codfw hosts are in tendril)
[13:52:10] <jynus>	 ok, that matches the downtimes
[13:52:58] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/509012/2/modules/profile/templates/mariadb/daily-snapshots-eqiad.cnf.erb -> any reason to have stop slave true on some but not on others? ie s8 has stop_slave: True but that is not the case with s2 or s3 or s5
[13:53:13] <jynus>	 at the moment is a guess
[13:53:25] <jynus>	 I belive it will be much faster on hosts with many writes
[13:53:39] <marostegui>	 ah ok, just checking if it was intentional :)
[13:53:41] <jynus>	 but it is a bit of a test
[13:54:10] <jynus>	 the good news is that I had left it like that but with the bug and still worked
[13:54:17] <jynus>	 so that should only make it faster
[13:54:22] <marostegui>	 haha
[13:54:31] <jynus>	 but we will see if it takes 4 hours or 24 hours
[13:54:36] <marostegui>	 fine by me, I am not touching any of those :)
[13:54:41] <marostegui>	 so just +1ed it
[13:54:53] <jynus>	 it is a bit of, let's gather data, then review
[13:55:02] <marostegui>	 I am going to grab a coffee, I need a break from writing perf reviews :)
[13:55:08] <jynus>	 please do
[13:55:18] * marostegui goes for a chai!
[14:56:18] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[14:57:44] <marostegui>	 db2105 finished the decompression and GTID worked, replication is flowing
[15:00:20] <marostegui>	 I will update the cheatsheet
[15:02:23] <marostegui>	 same with db2109 :)
[15:08:48] <marostegui>	 Worked like a charm, great job!
[15:23:04] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2112 doesn't show service tag in idrac - https://phabricator.wikimedia.org/T222845 (10RobH)
[15:24:34] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[15:24:42] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2112 doesn't show service tag in idrac - https://phabricator.wikimedia.org/T222845 (10Marostegui)
[15:39:04] <marostegui>	 recap of hosts
[15:39:08] <marostegui>	 db1081, already pooled in production
[15:39:16] <marostegui>	 db1138 ready with data but won't be pooled today
[15:39:26] <marostegui>	 db2105 and db2109 already on s3 but won't be pooled today
[15:39:33] <marostegui>	 db2112 HW maintenance
[15:39:37] <marostegui>	 db2114 HW maintenance
[15:40:04] <jynus>	 thanks
[15:59:31] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10RobH)
[15:59:44] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10RobH)
[16:06:54] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui)
[16:08:19] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui)
[16:08:45] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2114 hardware problem - https://phabricator.wikimedia.org/T222753 (10Papaul) 05Open→03Resolved @Marostegui @jcrespo  main board replacement complete on db2114. The problem has been resolved.  You can take over now.   System Board Fan1A 14% 3480 RPM 840 RPM N/A 480 RPM...
[16:10:06] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2114 hardware problem - https://phabricator.wikimedia.org/T222753 (10Marostegui) Thanks Papaul!