[05:59:27] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351#3196540 (10Marostegui) db2062 and db2069 are in the same state as yesterday (but with a lot less IOPS than after the initial peak) so that is...
[06:18:11] <wikibugs__>	 10DBA, 06Operations, 05codfw-rollout: Pool db2071? - https://phabricator.wikimedia.org/T163413#3196542 (10Marostegui)
[06:18:20] <wikibugs>	 10DBA, 06Operations, 05codfw-rollout: Pool new server db2071? - https://phabricator.wikimedia.org/T163413#3196556 (10Marostegui)
[07:00:21] <jynus>	 do I setup db1071?
[07:00:27] <marostegui>	 db2071?
[07:00:40] <jynus>	 yes
[07:00:42] <marostegui>	 If you want to, yes, it is already rebooted
[07:00:46] <marostegui>	 and looks good
[07:00:58] <marostegui>	 so I would say it can be a great addition to s1 :)
[07:03:47] <wikibugs>	 10DBA, 06Operations, 05codfw-rollout: Pool new server db2071 - https://phabricator.wikimedia.org/T163413#3196599 (10jcrespo)
[07:04:13] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3196601 (10jcrespo)
[07:04:16] <wikibugs>	 10DBA, 06Operations, 05codfw-rollout: Pool new server db2071 - https://phabricator.wikimedia.org/T163413#3196542 (10jcrespo)
[07:11:47] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3196615 (10Marostegui) db1040 oldimage table ready: ``` root@neodymium:~# mysql --skip-ssl commonswiki -hdb1040 -e "show cr...
[07:31:08] <jynus>	 db1067 ?
[07:31:31] <jynus>	 I was going to use it for cloning
[07:37:55] <marostegui>	 Oh
[07:38:01] <marostegui>	 I haven't done anything with it
[07:38:03] <marostegui>	 yet
[07:38:06] <marostegui>	 I can pick another host
[07:38:09] <marostegui>	 with no issues
[07:38:20] <marostegui>	 whatever you prefer
[07:38:23] <jynus>	 no, I already moved to db1080
[07:38:30] <marostegui>	 ok :)
[07:38:40] <jynus>	 but please send me the review so I can see it in advance
[07:38:56] <jynus>	 you do not even have to wait for me to +1
[07:39:03] <marostegui>	 ah sorry
[07:39:06] <marostegui>	 I didn't want to spam you
[07:39:08] <jynus>	 but add me so I can see it on email
[07:39:13] <jynus>	 no, I prefer to be spammed
[07:39:18] <marostegui>	 sure, you asked for it! :)
[07:39:20] <jynus>	 so that I know what is going on
[07:40:25] <jynus>	 are all enwiki hosts pending the revision changes?
[07:40:45] <jynus>	 because if they are, we may prefer to wait to clone?
[07:40:50] <marostegui>	 nope
[07:40:58] <marostegui>	 just 3 (including the master)
[07:41:38] <jynus>	 I think db1080 is done
[07:41:49] <marostegui>	 db1080 is done yes
[07:41:56] <marostegui>	 only pending: 65,67 and the master
[07:42:34] <marostegui>	 67 is only missing an index, the PK is correct
[07:44:03] <jynus>	 ok, I will deploy https://gerrit.wikimedia.org/r/349162 now
[07:44:14] <marostegui>	 ok!
[07:46:35] <marostegui>	 I am glad you didn't use db1067, because it has a mess of indexes :)
[07:46:48] <jynus>	 it was the old master
[07:46:52] <jynus>	 it makes sense
[07:47:26] <jynus>	 maybe we can run ANALYZE on revision at some point on all servers
[07:47:33] <jynus>	 on revision and logging
[07:47:47] <marostegui>	 yes
[07:47:57] <marostegui>	 I will run it on the hosts I am changing today
[07:48:17] <jynus>	 and then we can compare it with the top slow queries on codfw
[07:50:30] <jynus>	 https://phabricator.wikimedia.org/P5295
[08:03:44] <jynus>	 BTW, be careful with db2019 lag- it gets computed from itself
[08:04:01] <jynus>	 but it still replicates from s4-master-eqiad
[08:04:39] <marostegui>	 but it shouldn't get any, no?
[08:05:05] <jynus>	 not on icinga, but yes on SHOW SLAVE STATUS
[08:05:30] <jynus>	 or for example, if you restart the passive master, the active master complains
[08:05:32] <marostegui>	 I don't see it there either
[08:05:50] <jynus>	 ah, because the alter blocks replication
[08:05:58] <jynus>	 but not pt-heartbeat
[08:06:33] <marostegui>	 ah
[08:06:39] <jynus>	 I am just telling you this because I have generated alerts in the past with the passive datacenter
[08:06:43] <marostegui>	 yeah yeah
[08:06:51] <marostegui>	 good to know indeed
[08:07:05] <marostegui>	 I first altered the smallest of the two tables (11G) to see what would happen
[08:07:08] <marostegui>	 as I was doubting myself
[08:07:17] <marostegui>	 but it went well, so I went ahead with the big one
[08:07:22] <marostegui>	 but yes, good to know
[08:09:22] <marostegui>	 I am going to sleep a lot better once we have db2071 there serving
[08:10:51] <jynus>	 wow, cloning from SSDs to SSDs is FAST (I think this is the first time I do that)
[08:11:03] <marostegui>	 transfer rate?!
[08:11:05] <jynus>	 300MiB/s
[08:11:16] <marostegui>	 wow!
[08:11:18] <jynus>	 on a 1 Gb ethernet
[08:11:44] <jynus>	 because of compression and despite encryption
[08:20:51] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: Pool new server db2071 - https://phabricator.wikimedia.org/T163413#3196712 (10Marostegui) db2072 is also now ready to be used if needed. All went fine after rebooting it.
[08:20:53] <marostegui>	 jynus ^
[08:27:19] <jynus>	 why do they have puppet default stuff?
[08:27:28] <jynus>	 is there are role applied by default?
[08:27:38] <marostegui>	 I am running it manually :)
[08:27:51] <marostegui>	 to be able to ssh to it and all that
[08:27:51] <jynus>	 no
[08:28:04] <jynus>	 what I mean is, what do you run?
[08:28:06] <jynus>	 puppet run?
[08:28:08] <marostegui>	 ah, just puppet
[08:28:09] <marostegui>	 yes
[08:28:27] <jynus>	 because it is not part of a spare role
[08:28:45] <marostegui>	 Ah, I get what you mean
[08:29:03] <jynus>	 but it gets added to icinga, etc.
[08:29:12] <jynus>	 and sets up ssh accounts
[08:29:30] <marostegui>	 true true
[08:29:42] <jynus>	 how is that possible?
[08:29:46] <jynus>	 specially the accounts
[08:30:14] <marostegui>	 ah
[08:30:46] <marostegui>	 No, nevermind
[08:30:52] <marostegui>	 that is weird yes
[08:31:06] <jynus>	 is it defined elsewhere and we are duplicating the role?
[08:31:16] <marostegui>	 I cannot find it on the repo
[08:31:24] <jynus>	 is there an automagically spare role?
[08:31:32] <marostegui>	 not even by grepping for: "|72" and stuff like that
[08:31:41] <marostegui>	 The spare role at least months ago, you had to manually add it
[08:31:43] <marostegui>	 to make it spare
[08:32:39] <marostegui>	 the only commits in gerrit for db2071 are the ones you added
[08:32:42] <marostegui>	 and db2072 has none
[08:34:17] <jynus>	 but see it on icinga
[08:34:36] <marostegui>	 yes
[08:34:38] <marostegui>	 it just came up
[09:09:54] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: Pool new server db2071 - https://phabricator.wikimedia.org/T163413#3196888 (10Marostegui) More servers are now online and ready to be used if needed, I will post it on the original tracking task (T162159) so I don't hijack this one.
[09:13:31] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3196923 (10Marostegui) @Papaul the following servers were ready to get puppet enabled and all that, so I did so, and rebooted them  db2071 db2072 db2073 db2075 db2076 db2079  db2...
[09:16:52] <moritzm>	 marostegui: db2071 will be the first mysql server with Linux 4.9, I don't expect any problems, but if something is weird, please let me know
[09:17:33] <marostegui>	 moritzm: will do, thanks :)
[09:33:58] <jynus>	 will we have enough space on db1040?
[09:35:01] <marostegui>	 I checked one of the hosts with file per table, and the table was 135G
[09:35:08] <marostegui>	 *is
[09:35:40] <marostegui>	 There is still 163G to go on db1040 and the alter has been running for 2h already
[09:35:42] <marostegui>	 so I think we will
[09:36:01] <jynus>	 do you know how long it will take, based on the other alters?
[09:36:12] <marostegui>	 the other alters were taking around 4-5h
[09:36:18] <jynus>	 good
[09:36:36] <jynus>	 https://img.memesuper.com/08fdae7898ea66c240e710db5b8069d1_yes-let-the-hate-flow-through-you-emperor-palpatine-quickmeme-sidious-good-meme_494-358.jpeg
[09:36:43] <marostegui>	 xdddddddd
[09:39:11] <marostegui>	 [09:39:04] marostegui@db2071:~$ df -hT /srv/
[09:39:11] <marostegui>	 Filesystem            Type  Size  Used Avail Use% Mounted on
[09:39:11] <marostegui>	 /dev/mapper/tank-data xfs   3.6T  1.3T  2.4T  34% /srv
[09:39:13] <marostegui>	 almost done?
[09:39:16] <marostegui>	 :)
[09:39:24] <jynus>	 almost
[11:17:53] <jynus>	 marostegui, I will go to have lunch, I will write to ticket later
[11:18:04] <marostegui>	 jynus: sure thing!
[11:18:06] <jynus>	 or you can do it if you want to do it before
[11:18:08] <marostegui>	 enjoy
[11:18:18] <marostegui>	 i will head for food in 5 minutes too
[11:18:37] <jynus>	 db2071 is working but replicationg from eqiad master
[11:18:53] <jynus>	 I will swich it now to codfw master
[11:19:14] <marostegui>	 cool, will you stop eqiad master and get the position and then change it to the same one?
[11:19:23] <marostegui>	 that was not very well explained, but I think you get what I meant
[11:19:25] <jynus>	 https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=db2071&from=now-3h&to=now
[11:19:33] <jynus>	 CHANGE MASTER TO with GTIDs :-)
[11:19:45] <marostegui>	 oh yes, we have gtids!! 
[11:19:46] <jynus>	 automatic, no need for the script
[11:19:48] <marostegui>	 I forgot
[11:19:52] <marostegui>	 :)
[11:19:55] <jynus>	 that is why I started as is
[11:19:57] <marostegui>	 is uit using SSL already?
[11:20:13] <jynus>	 of course, no cross-dc should do it without ssl
[11:20:21] <marostegui>	 :)
[11:20:28] <jynus>	 all replication channels use it, in fact
[11:20:46] <jynus>	 we should enforce it when we setup the better SSL monitoring
[11:24:11] <jynus>	 STOP SLAVE; CHANGE MASTER TO MASTER_HOST='db2016.codfw.wmnet'; START SLAVE;
[11:24:18] <jynus>	 it worked seamlessly
[11:24:25] <marostegui>	 <3
[11:24:31] <marostegui>	 gtid! \o/
[11:25:34] <jynus>	 we can pool it already
[11:25:57] <marostegui>	 i can do it before going for lunch
[11:26:04] <marostegui>	 with some small weight
[11:26:07] <jynus>	 yeah
[11:26:15] <jynus>	 it hasn't finished loading the buffer pool
[11:26:20] <jynus>	 yet
[11:26:22] <marostegui>	 I will give it 20 or so
[11:26:28] <jynus>	 but I get it will be faster now
[11:26:36] <jynus>	 than the other servers
[11:26:40] <marostegui>	 for sure
[11:28:36] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/349202/
[11:28:58] <jynus>	 are main or as api?
[11:29:03] <marostegui>	 both
[11:29:07] <marostegui>	 20 as main and 1 as api
[11:29:15] <jynus>	 good
[11:30:00] <marostegui>	 we can probably leave it with 50 as main and then more api than the others
[11:30:12] <marostegui>	 and maybe even depool one of the others to run analyze over the weekend
[11:30:45] <jynus>	 I am starting 10.0.30 on 1080
[11:30:56] <marostegui>	 cool
[11:31:32] <jynus>	 I put mysql_upgrade on path on the latest packages
[11:31:41] <marostegui>	 yeah
[11:31:53] <marostegui>	 and i loved it when i set up the last s5 hosts 
[11:35:01] <marostegui>	 deployed
[11:38:38] <marostegui>	 https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=db2071
[11:40:37] <marostegui>	 The new server is filesorting too
[11:41:47] <marostegui>	 https://phabricator.wikimedia.org/P5293#28339
[11:41:47] <jynus>	 it wasn't on the original one
[11:42:09] <jynus>	 but now it does
[11:42:20] <marostegui>	 which one did you use in the end?
[11:42:22] <marostegui>	 db1080?
[11:42:25] <jynus>	 yes
[11:42:29] <marostegui>	 because it does filesort too :|
[11:42:35] <jynus>	 it didn't before
[11:42:37] <marostegui>	 maybe after restarting mysql the plan goes crazy?
[11:42:45] <marostegui>	 *optimizer
[11:45:30] <marostegui>	 at least a server with ssd filesorting is still better than the other two with more weight: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&orgId=1&var-server=db2071&var-network=eth0
[11:46:04] <marostegui>	 going to get lunch
[12:31:58] <marostegui>	 maybe we can try analazying table on db1080, then checking if the filesort is gone, and then restarting mysql to see if we can reproduce that error
[12:41:11] <_joe_>	 so I just saw this in #-staff
[12:41:12] <_joe_>	 14:07 < Trizek> I have a user who wants to open this link: https://commons.wikimedia.org/w/index.php?title=Special:NewFiles&dir=prev&offset=20160520124303&limit=500. I
[12:41:16] <_joe_>	                 don't know how the offset value has been set, but it is apparently what is broken.
[12:41:19] <_joe_>	 14:08 < Trizek> However, https://commons.wikimedia.org/w/index.php?title=Special:NewFiles&dir=prev&offset=20060730182009&limit=500 works. I'm confused.  Any idea?
[12:41:35] <_joe_>	  &offset=20160520124303 seems like the kind of parameter that causes a very slow query
[12:41:52] <_joe_>	 am I wrong?
[12:43:44] <wikibugs__>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351#3197434 (10Marostegui) db2071 has now been serving traffic for around 1 hour:  https://grafana.wikimedia.org/dashboard/file/server-board.json?...
[12:45:28] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351#3197438 (10Marostegui) Maybe with this extra server, we can now depool one of the other "old" ones and let them run analyze over the weekend,...
[12:46:06] <marostegui>	 _joe_: I am not completely sure how that works, but it doesn't look like somthing nice to the DB indeed, no
[12:47:42] <wikibugs__>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: Pool new server db2071 - https://phabricator.wikimedia.org/T163413#3197446 (10Marostegui) I have pooled db2071 with the same main traffic as the other api servers (50) but with weight 2 in API, instead of 1 as the other servers. We'll see how it goe...
[12:50:53] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3197453 (10Marostegui) db1040 is done for the image table too: ``` root@neodymium:~# mysql --skip-ssl -hdb1040 commonswiki...
[12:58:17] <wikibugs>	 10DBA, 07Epic, 13Patch-For-Review, 05codfw-rollout: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) - https://phabricator.wikimedia.org/T155099#3197496 (10Marostegui)
[12:58:20] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197497 (10Marostegui)
[12:58:23] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3197495 (10Marostegui) 05Open>03Resolved
[13:16:29] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197527 (10jcrespo) > This tracking task could be converted into a project (eg #wmf-schema-change) but I'll leave that to @Springle and @jcrespo.  I am about to deprecate this tracking...
[13:17:09] <wikibugs>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197528 (10jcrespo)
[13:23:40] <marostegui>	 db1040 alter finished, we'll see how long it takes to reclaim the disk space back after the alter
[13:24:24] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3197549 (10Addshore)
[13:24:28] <wikibugs__>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 06TCB-Team, and 2 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3087424 (10Addshore) 05Open>03declined
[13:25:09] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 06TCB-Team, and 2 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3087424 (10Addshore) Looks like we are not going to do this, as it will not be needed as wl_id should be added d...
[13:26:06] <wikibugs__>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 06TCB-Team, and 2 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3197568 (10Marostegui) I still think it would be nice to have that feature there if needed
[13:27:23] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3197573 (10Addshore)
[13:27:29] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 06TCB-Team, and 2 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3197571 (10Addshore) 05declined>03Open >>! In T160062#3197568, @Marostegui wrote: > I still think it would b...
[13:27:42] <addshore>	 :)
[13:27:49] <marostegui>	 thanks!
[13:28:00] <marostegui>	 if it is already done, i think it would be nice to have. You never know what the future brings!
[13:28:21] <addshore>	 very true :)
[13:29:01] <addshore>	 in a dream world i guess you could mark individual tables in mediawiki as read-=only / to be handled correctly ;)
[13:30:09] <marostegui>	 haha that would be ideal, you think we can have that by 2030? :)
[13:30:26] <addshore>	 who knows! :P
[13:32:39] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3197582 (10jcrespo)
[13:34:30] <wikibugs__>	 10DBA: Update change tag indexes - https://phabricator.wikimedia.org/T42867#3197586 (10jcrespo)
[13:34:32] <wikibugs>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197587 (10jcrespo)
[13:34:55] <wikibugs__>	 10DBA: Update change tag indexes - https://phabricator.wikimedia.org/T42867#467053 (10jcrespo) Probably done, but still pending to check.
[13:37:29] <wikibugs>	 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#553002 (10jcrespo) Probably pending to apply. If that is the case, it should be added to the #blocked-on-schema-change project. If done within a week, it may be able to done quicky, otherwise, it may have to wait another...
[13:37:38] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197594 (10Anomie)
[13:37:45] <wikibugs>	 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#3197596 (10jcrespo)
[13:37:47] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197597 (10jcrespo)
[13:38:43] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3197600 (10Anomie)
[13:40:24] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#3197607 (10jcrespo)
[13:41:57] <wikibugs>	 10DBA, 07Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191#553101 (10jcrespo) Probably done? Needs checking. Otherwise, adding the #blocked-on-schema-change.
[13:42:21] <wikibugs__>	 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#853220 (10jcrespo)
[13:42:23] <wikibugs>	 10DBA, 07Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191#3197611 (10jcrespo)
[14:55:46] <jynus>	 interesting: http://mysqlhighavailability.com/more-metadata-is-written-into-binary-log/
[14:58:08] <marostegui>	 that is nice
[14:58:16] <marostegui>	 I wonder how much is that going to increase the binlog size
[14:59:40] <marostegui>	 I read this yesterday: https://mydbops.wordpress.com/2017/04/13/binlog-expiry-now-in-seconds-mysql-8-0/
[14:59:43] <marostegui>	 which is nice too
[15:00:10] <jynus>	 bah
[15:00:16] <jynus>	 I liked more the percona way
[15:00:24] <jynus>	 expiratino based on size
[15:00:40] <marostegui>	 yes, that is a great feature
[15:01:02] <marostegui>	 is it not in mysql 8? :(
[15:02:18] <wikibugs>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197864 (10Nirmos)
[15:02:29] <jynus>	 not really that important, that is one that can be simulated manually
[15:04:43] <marostegui>	 I don't get at all what that ticket means, that that user got his watchlist list reseted I guess?
[15:05:24] <wikibugs>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197864 (10jcrespo) I can give the user (and only himself) a list of watched articles 24 hours ago, and maybe he can import it, but we will not recover it automatically, as that is not easy or practical to do.
[15:06:19] <jynus>	 he wants a backup- I should have denied it, because if we offer it once, other people will ask
[15:06:28] <jynus>	 and we cannot attend 1000000 users
[15:06:56] <marostegui>	 Yeah I am seeing that
[15:07:12] <marostegui>	 I was doing the recovery exercise myself and I think I got the data :)
[15:07:20] <marostegui>	 for learning purproses
[15:07:31] <jynus>	 I just weas feelining like working until midnight today
[15:07:46] <marostegui>	 xddddd
[15:07:51] <jynus>	 actually, learning the schema meaning
[15:08:03] <jynus>	 is something that probably will eventually will help you
[15:08:10] <marostegui>	 Yeah, that is why I did it
[15:08:17] <marostegui>	 And i think I got it :)
[15:08:19] <jynus>	 but note that is private data
[15:08:21] <marostegui>	 For this ticket I mean
[15:08:24] <marostegui>	 Oh yes, of course
[15:08:29] <jynus>	 we cannot publish it
[15:08:40] <jynus>	 for example, imagine the request is fake
[15:08:49] <marostegui>	 No no, I was just doing it for me to resolve the ticket in my mind
[15:08:58] <marostegui>	 Yeah, how can we verify the user actually?
[15:09:04] <jynus>	 user has to provide a private way to do it
[15:09:24] <jynus>	 not sure how, wiki admins maybe can figure out a way
[15:11:21] <jynus>	 I did mysql -h dbstore1002.eqiad.wmnet svwiki -e "SELECT wl_namespace, wl_title FROM watchlist JOIN user ON user_id = wl_user WHERE user_name = 'I99pema' "
[15:11:28] <marostegui>	 yes
[15:11:30] <marostegui>	 Same thing :)
[15:11:44] <jynus>	 but if it has only 20 items may be too late
[15:12:51] <marostegui>	 I see 46, but yes
[15:12:56] <marostegui>	 we will see what he says
[15:14:09] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197894 (10I99pema) 05Open>03Resolved a:03I99pema A list would be of great help!
[15:14:18] <marostegui>	 :|
[15:14:32] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197897 (10jcrespo) 05Resolved>03Open
[15:14:35] <marostegui>	 User Since
[15:14:35] <marostegui>	 Thu, Apr 20, 17:10 (3 m, 41 s)
[15:14:47] <jynus>	 lets check the user is joined with the global account
[15:15:05] <jynus>	 then we can maybe paste it on phab with permissions only for NDA and him
[15:15:49] <jynus>	 I would ask for the user to indetify itself
[15:15:53] <marostegui>	 yes
[15:15:57] <jynus>	 on wiki
[15:16:11] <marostegui>	 I would ask him to edit his wiki page
[15:16:12] <marostegui>	 exactly
[15:16:25] <jynus>	 saying he is the same user on phabricator
[15:16:51] <jynus>	 do you want to handle that?
[15:17:00] <jynus>	 or Do I?
[15:17:13] <marostegui>	 I will
[15:18:13] <jynus>	 export it in plain text, it will be easier (mysql instead of mysqldump)
[15:18:24] <marostegui>	 sure
[15:18:38] <jynus>	 oph, you did
[15:18:45] <jynus>	 the .sql was fake
[15:18:48] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197864 (10Marostegui) Hello @I99pema, we first need to make sure that you are who you said you are before handling private data to you. Can you please edit your personal wikipage affirming that you are the same person on sv.wikipedia.org...
[15:18:57] <marostegui>	 yes, I tend to use .sql for all the stuff coming out from mysql
[15:19:19] <jynus>	 buuuuuh
[15:19:27] <marostegui>	 xddddddd
[15:26:30] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197956 (10I99pema)  Like this? https://sv.wikipedia.org/w/index.php?title=Användare%3AI99pema&type=revision&diff=39623432&oldid=38428127
[15:28:00] <marostegui>	 jynus ^ that looks good to me
[15:28:03] <marostegui>	 same user
[15:28:07] <marostegui>	 and edited with the same user
[15:29:58] <jynus>	 technically, he linked phab to mediawiki, and not the other way round, but let's give it the ok
[15:32:32] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197972 (10Marostegui) Thanks This is what you had in your watchlist: https://phabricator.wikimedia.org/P5297
[15:34:44] <wikibugs>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3197976 (10I99pema) No, that's what has been added to the list after it was deleted earlier today
[15:35:06] <jynus>	 oh, you used dbstore1002?
[15:35:13] <marostegui>	 just realised :)
[15:36:30] <marostegui>	 he's got 10k ones :)
[15:36:45] <jynus>	 you can add; SELECT max(rev_timestamp) FROM revision; or SELECT max(ts) FROM heartbeat.heartbeat
[15:41:15] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3198022 (10Marostegui) Sorry, went to the wrong server. Check again the new update.
[15:59:43] <wikibugs__>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3198109 (10I99pema) Thats the correct one. I have now restored my Watch list and the case can be closed.  Thank you so much for your help and quick response!
[16:00:26] <wikibugs>	 10DBA: I99pema reset watchlist - https://phabricator.wikimedia.org/T163450#3198117 (10Marostegui) 05Open>03Resolved a:05I99pema>03Marostegui Great!
[16:06:25] <wikibugs>	 10DBA, 13Patch-For-Review: Reclone db1068 to become a slave in s4 - https://phabricator.wikimedia.org/T163110#3198153 (10Marostegui) Server is recloned. Catching up SSL enabled GTID using: slave_pos
[16:07:29] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3198158 (10Papaul) @Marostegui Thanks for for update.
[16:08:12] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3198161 (10Papaul)
[16:15:21] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3198202 (10Marostegui) db1065 is done:  ``` root@neodymium:~# mysql --skip-ssl -hdb1065 enwiki -e "show create table revision\G" *************************** 1. row ****...
[16:25:18] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup 22 DB servers - https://phabricator.wikimedia.org/T162159#3198283 (10Papaul) @Marostegui none of the systems were ready.
[16:35:58] <wikibugs__>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351#3198322 (10Marostegui) Just for the record, I finished an alter table on the revision table on db1065, and it is not filesorting.  ``` root@db...
[17:38:16] <chatter29>	 hey guys
[18:16:20] <wikibugs>	 10DBA: Network maintenance on row D (databases) - https://phabricator.wikimedia.org/T162681#3198923 (10Marostegui) db1068 has been promoted to master (we had to do it kind unexpectedly today) on s4 - it is affected by the recabling but no for the server move. So that is good.
[18:16:48] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3198924 (10Marostegui) db1068 is now s4 master.
[18:17:58] <wikibugs__>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3198929 (10Marostegui)
[18:21:00] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3198937 (10Marostegui)
[18:22:40] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3153482 (10Marostegui)
[18:22:43] <wikibugs__>	 10DBA, 13Patch-For-Review: Reclone db1068 to become a slave in s4 - https://phabricator.wikimedia.org/T163110#3198940 (10Marostegui) 05Open>03Resolved db1068 is now serving as a master see: T162133
[18:24:18] <wikibugs>	 10DBA, 13Patch-For-Review: Reclone db1068 to become master in s4 - https://phabricator.wikimedia.org/T163110#3198956 (10Marostegui) a:03jcrespo
[18:48:30] <wikibugs>	 10DBA, 13Patch-For-Review: Reclone db1068 to become master in s4 - https://phabricator.wikimedia.org/T163110#3199105 (10jcrespo) For the record, I ran:  ``` ./repl.pl --switch-child-to-sibling --parent=db1040.eqiad.wmnet --child=dbstore1002.eqiad.wmnet --child-set="default_master_connection='s4'" /repl.pl --sw...
[18:49:12] <jynus>	 db1047
[21:49:39] <wikibugs__>	 10DBA, 06Operations, 10Phabricator: Intermitten outage on phabricator, needs investigation - https://phabricator.wikimedia.org/T163507#3199906 (10Aklapper) p:05Unbreak!>03High This is intermittent so I don't see why this should be Unbreak Now
[21:50:02] <wikibugs>	 10DBA, 06Operations, 10Phabricator: Intermittent DB connectivity problem on phabricator, needs investigation - https://phabricator.wikimedia.org/T163507#3199908 (10Aklapper)
[23:58:26] <wikibugs__>	 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#3200317 (10tstarling) {P5302}  Tested locally.