[05:37:20] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3208743 (10Marostegui) db1052 eqiad master is done: ``` root@neodymium:~# mysql --skip-ssl enwiki -e "show create table revision\G" -hdb1052 ***************************...
[05:53:27] <wikibugs__>	 07Blocked-on-schema-change, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3208755 (10Marostegui) db1082 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb1082 wikidatawiki -e "sho...
[05:54:16] <wikibugs>	 10DBA, 10Wikidata, 13Patch-For-Review, 07Schema-change: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" on "wb_terms" table - https://phabricator.wikimedia.org/T163548#3208756 (10Marostegui) db1082 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb1082 wikidatawiki -e "show crea...
[06:50:15] <marostegui>	 any objection if I stop mysql on s7's master to do a normal transfer to db1062? the mysqldump died over night and it is just faster to stop it for 2 hours and copy all the content
[06:50:20] <marostegui>	 s7 eqiad master, obviously
[07:10:15] <jynus>	 ok to me
[07:11:33] <marostegui>	 oki
[07:11:35] <marostegui>	 will start in a bit
[07:17:36] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3208863 (10Marostegui) Thanks @jcrespo for testing it! I have altered all **s3** wikis (that have watchlist table)...
[07:31:00] <jynus>	 so I will do the master failover of s5?
[07:31:23] <marostegui>	 maybe s6 instead? s5 is running an alter now
[07:31:57] <marostegui>	 I was silencing s2, I haven't started yet with it, I can do the watchlist alter on s6 if you want do do s2 master switch?
[07:34:44] <jynus>	 ok, I will do s2 instead
[07:34:54] <marostegui>	 ok
[07:34:57] <marostegui>	 I will do s6 then 
[07:35:01] <marostegui>	 s2 is silenced
[07:37:00] <jynus>	 we can silence all of eqiad
[08:02:16] <jynus>	 I will start moving things around on s2- check sal for updates
[08:03:18] <marostegui>	 ok!
[08:03:19] <marostegui>	 thanks
[08:34:44] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3208971 (10Alchimista) Thanks @Dereckson, @jcrespo .
[08:40:27] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3208992 (10Marostegui) s6 is done. db1093 already had the correct schema in all the databases so it failed. I skipp...
[08:42:30] <jynus>	 if a slave had the schema already applied- its index may have drifted and it will likely fail when more edits are received
[08:43:08] <jynus>	 that was the error you got with tags long time ago in the passive dc
[08:43:55] <marostegui>	 yeah, I am going to reclone it once the others have caught up
[08:44:08] <jynus>	 no need for a full reclone
[08:44:14] <marostegui>	 yeah, i meant that table
[08:44:15] <jynus>	 just the watchlist table
[08:44:17] <marostegui>	 yeah
[09:08:30] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209057 (10Addshore) >>! In T130067#3208863, @Marostegui wrote: > Thanks @jcrespo for testing it! > I have altered...
[09:09:32] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209058 (10Marostegui) >>! In T130067#3209057, @Addshore wrote: >>>! In T130067#3208863, @Marostegui wrote: >> Than...
[09:11:39] <jynus>	 so s2 is clear for watchlist
[09:11:47] <jynus>	 but notice the new master!
[09:11:57] <marostegui>	 yep
[09:11:58] <marostegui>	 :)
[09:11:59] <marostegui>	 thanks!
[09:34:56] <jynus>	 can I do s6 or any other?
[09:35:21] <marostegui>	 yep, take s6
[09:35:26] <marostegui>	 I am going to do s2 in a bit
[09:36:02] <jynus>	 ok, will do master db1050 -> db1061
[09:36:52] <marostegui>	 yep, db1061 is the one
[09:38:58] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209105 (10Marostegui) db1093 watchlist table from frwiki, jawiki and ruwiki has been re-imported.
[09:40:01] <jynus>	 there is lag on some replicas of s6, are schema changes or imports still ongoing?
[09:40:25] <marostegui>	 there is no lag anuymore, if it was db1088 and db1093 I was reimporting the table
[09:40:28] <marostegui>	 but that is done
[09:40:35] <jynus>	 ok, thanks
[09:40:36] <marostegui>	 I don't see anylag anymore
[09:40:40] <marostegui>	 but double check :)
[09:40:47] <jynus>	 yes, it was a few minutes ago
[09:40:51] <jynus>	 no problem now
[09:40:54] <marostegui>	 :)
[09:40:58] <jynus>	 ok, about to do s6
[09:41:02] <marostegui>	 great
[09:41:51] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209108 (10Marostegui) s4 is done db1091 had the schema already there, so I will reimport its table once the other...
[10:12:53] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209143 (10Marostegui) >>! In T130067#3209108, @Marostegui wrote: > s4 is done > db1091 had the schema already ther...
[10:18:46] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3209168 (10Marostegui) db1095 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb1095 -e "show create table revision\G" enwiki ****...
[10:33:25] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209228 (10Marostegui) s2 is done. db1090 needs to get the table reimported.
[10:36:03] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 13Patch-For-Review: Recreate a wiki for Wikimedia Portugal - https://phabricator.wikimedia.org/T126832#3209237 (10Dereckson)
[10:38:32] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3209240 (10Marostegui)
[10:49:28] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 06TCB-Team, and 3 others: Add wl_timestamp to the watchlist table - https://phabricator.wikimedia.org/T125991#2002327 (10daniel) I think we want this field, but I'm worried that the intended semantics is still unclear.  In the patch by addshore, the...
[11:04:23] <jynus>	 as I did it old scool, I am going to reenable gtid on all eqiad slaves now
[11:04:28] <jynus>	 of s6
[11:05:18] <marostegui>	 great job
[11:05:27] <marostegui>	 all the switchovers, just amazing
[11:05:49] <jynus>	 nope, I just broke things
[11:06:12] <marostegui>	 you didn't it was broken before
[11:06:21] <jynus>	 I don't think so
[11:08:06] <jynus>	 we have to review gtid on masters on switchback
[11:09:50] <jynus>	 things should be ok now
[11:10:43] <jynus>	 I do not see errors on s6
[11:10:51] <jynus>	 and only see master not found on s7
[11:10:57] <jynus>	 and some lag on s2
[11:11:02] <marostegui>	 I just brought up s7 master :)
[11:11:02] <jynus>	 which was expected
[11:11:05] <marostegui>	 I was updating the ticket
[11:11:15] <jynus>	 yeah, I am reviewing everthing
[11:11:25] <wikibugs>	 10DBA: Reclone db1062 from db1041 (s7 master) - https://phabricator.wikimedia.org/T163665#3209300 (10Marostegui) db1062 has been recloned from db1041. db1041 has been brought back and db1062 is replicating from it
[11:11:27] <jynus>	 because with the downtimes
[11:11:33] <marostegui>	 yeah
[11:11:39] <jynus>	 we may be missing some errors
[11:11:43] <marostegui>	 I am revieing tendril every now and then
[11:11:51] <jynus>	 I will downtime db1022
[11:11:51] <marostegui>	 to make sure what is down is what i expect to be down
[11:11:54] <marostegui>	 ok
[11:11:58] <jynus>	 but I will not even bother to fix it
[11:12:01] <marostegui>	 s2 slaves are coming back in a bit
[11:12:05] <jynus>	 just decomission
[11:12:06] <marostegui>	 And I will start with s1
[11:12:13] <marostegui>	 we might want to decomission db1040 too
[11:12:16] <jynus>	 waht about me?
[11:12:24] <jynus>	 what is "free" ?
[11:12:41] <jynus>	 I have pending s5 and s7
[11:12:55] <marostegui>	 I would wait for those to catch up
[11:12:59] <jynus>	 ok
[11:13:01] <marostegui>	 s5 is doing the alter
[11:13:04] <jynus>	 so I will go have lunch
[11:13:07] <marostegui>	 go :)
[11:13:08] <jynus>	 now
[11:13:16] <jynus>	 and do those after lunch
[11:13:19] <marostegui>	 great
[11:14:13] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209302 (10Marostegui) db1090 has been reimported from db1076
[11:15:29] <jynus>	 I am going to file a ticket for db1022
[11:15:36] <jynus>	 so we do not lose track of it
[11:15:51] <marostegui>	 ok!
[11:16:06] <wikibugs>	 10DBA, 05codfw-rollout: db1022 broke while changing topology on s6- evaluate if to fix or directly decomission - https://phabricator.wikimedia.org/T163778#3209308 (10jcrespo)
[11:16:23] <jynus>	 I had the good idea of starting with the least important hosts first!
[11:17:10] <jynus>	 but I have the suspicion this is a race condition similar to the multi-source replication one
[11:17:24] <marostegui>	 could be yes
[11:17:25] <jynus>	 someting got written to the slave or the master out of band
[11:17:33] <jynus>	 and now it misses coordinates
[11:17:58] <marostegui>	 when the multi source had the issue, it was displaying the coordinates it was expecting
[11:18:01] <marostegui>	 just saying
[11:18:07] <marostegui>	 but I agree, it looks very similar
[11:18:32] <jynus>	 Got fatal error 1236 from master when reading data from binary log: 'Could not find GTID state
[11:18:53] <jynus>	 the primary key errors is me getting nervous and trying to revert
[11:19:27] <marostegui>	 ah
[11:19:28] <marostegui>	 ok ok
[11:19:52] <jynus>	 I am suspecting meariadb gtid implementation is broken
[11:20:03] <marostegui>	 for multisource it clearly is
[11:20:05] <jynus>	 mysql tracks ranges
[11:20:08] <marostegui>	 so for this, I wouldn't be surprised
[11:20:11] <marostegui>	 if it is too
[11:20:12] <jynus>	 while mariadb tracks positions
[11:20:33] <jynus>	 when you change a master on mysql you tell it to "assume everhing on this range has been done"
[11:20:40] <marostegui>	 maybe for the new switchovers we should do reset master?
[11:20:44] <jynus>	 while in mariadb you tell, sttart from this position
[11:20:45] <marostegui>	 if there is still no slave attached?
[11:21:07] <jynus>	 but in the other bug, reset master didn't work?
[11:21:24] <marostegui>	 it was reset master + a buinch of things, like deleting the file, truncating the gtid table etc
[11:21:30] <jynus>	 pffff
[11:21:36] <marostegui>	 i know XD
[11:21:44] <jynus>	 well, to be fair
[11:21:52] <jynus>	 if that makes it work for multisource
[11:22:00] <jynus>	 we could setup an intermediate slave?
[11:22:20] <marostegui>	 what for?
[11:22:24] <jynus>	 maybe we already had this conversation
[11:22:45] <jynus>	 to run reset master there- but it doesn matter, there is already a master where that is not happening
[11:23:35] <marostegui>	 ah
[11:23:51] <marostegui>	 But I am not sure what happens what happens with the primary master and its gtid coordinates
[11:23:54] <marostegui>	 and past events
[11:23:56] <jynus>	 yes
[11:24:04] <jynus>	 that is the whole point of this maybe broken
[11:24:13] <jynus>	 this == mariadb gtid replication
[11:24:22] <marostegui>	 yeah
[11:24:41] <jynus>	 at this point gtid is not useful
[11:24:48] <marostegui>	 no, it is not
[11:24:52] <jynus>	 except for transactional replication control
[11:24:54] <marostegui>	 i mean it is, but it is very flaky
[11:24:59] <jynus>	 which is tight to it
[11:25:05] <jynus>	 *tied
[11:33:09] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3209341 (10Marostegui)
[11:53:06] <wikibugs__>	 10DBA, 06Labs: archive/archive_userindex is not filled in eswiki_p - https://phabricator.wikimedia.org/T133251#3209362 (10Superzerocool) After one year, is there any solution or just is a minor/lowest issue?
[12:13:47] <wikibugs>	 10DBA: Reclone db1062 from db1041 (s7 master) - https://phabricator.wikimedia.org/T163665#3209371 (10Marostegui) 05Open>03Resolved a:03Marostegui db1062 has caught up SSL is running GTID is running binlog format is in MIXED
[12:13:50] <wikibugs__>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3209374 (10Marostegui)
[12:31:24] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209591 (10Marostegui) s5 is done. db1092 needs its table reimported.
[12:45:21] <jynus>	 going for s7 master switch
[12:45:33] <jynus>	 to db1062
[12:46:15] <marostegui>	 great
[12:46:21] <marostegui>	 once you are done I will do the alter there
[13:06:25] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209756 (10Marostegui) >>! In T130067#3209591, @Marostegui wrote: > s5 is done. db1092 needs its table reimported....
[13:19:34] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209783 (10Marostegui) I have altered labstestweb2001 and silver too to avoid the issues encountered at: T155605 ye...
[13:32:48] <wikibugs>	 07Blocked-on-schema-change, 10DBA: Apply change_tag and tag_summary primary key schema change to Wikimedia wikis - https://phabricator.wikimedia.org/T147166#3209821 (10Marostegui) As well as we did here: T130067#3206224 I have started to deploy this on a small wiki on s3, to see how it goes. I have chosen `etw...
[13:33:39] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209823 (10Marostegui) s1 is done. db1089 needs its table reimported, I will do so once the slaves have caught up
[13:36:33] <marostegui>	 going for lunch now :|
[14:38:43] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3209993 (10Marostegui) dbstore1001 just got the change for etwiki fine (which was done yesterday). So tomorrow I wi...
[14:46:44] <jynus>	 I got the same error when changing the master of db1062
[14:47:02] <jynus>	 happily, I had not started the slave sql thread yet, and I had stopped the master everywhere
[14:48:16] <marostegui>	 so db1062 had the error?
[14:48:23] <jynus>	 it or its master
[14:48:27] <jynus>	 who knows
[14:48:34] <marostegui>	 could be, because it was just recloned
[14:48:36] <marostegui>	 from it
[14:51:30] <jynus>	 It least this time I was prepared for the failure
[14:53:32] <marostegui>	 haha
[14:53:36] <marostegui>	 that is half of the way
[14:53:37] <jynus>	 itr true
[14:53:50] <jynus>	 I tested to make sure it was not me
[14:53:57] <jynus>	 but I had the plan B ready
[14:54:45] <jynus>	 I think there is now not a slave with partitioning on s7
[14:55:02] <jynus>	 but partitioning was breaking more than it solved there
[14:55:32] <marostegui>	 i hate those snowflakes :(
[14:58:59] <jynus>	 if you want to give s7 a second look, but I think it is ok now
[14:59:06] <marostegui>	 is it all done now?
[14:59:09] <jynus>	 (except for dbstore1002)
[14:59:11] <jynus>	 1001
[14:59:29] <jynus>	 which was missing from yesterday
[15:01:48] <jynus>	 so I am only missing s5
[15:01:52] <jynus>	 is that ready?
[15:02:07] <marostegui>	 nope, the alter still going
[15:02:11] <marostegui>	 I would leave it till tomorrow 
[15:02:43] <marostegui>	 can I go ahead and alter s7?
[15:02:44] <wikibugs__>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3210073 (10Marostegui)
[15:02:46] <jynus>	 so are you doing an alter one at a time or how?
[15:02:55] <jynus>	 can you give more more details?
[15:03:08] <marostegui>	 I am altering 3 hosts at the same time
[15:03:16] <marostegui>	 It should be almost finished
[15:03:22] <marostegui>	 by tomorrow (all the hosts, I mean)
[15:03:34] <marostegui>	 (eqiad hosts)
[15:03:48] <jynus>	 when you finish working, can you give me a list of X: done Y: in progres Z: pending
[15:03:52] <marostegui>	 yes
[15:03:54] <marostegui>	 :)
[15:04:29] <jynus>	 Yes, as far as I can see, s7 is ok
[15:04:46] <marostegui>	 yes, I can see it working fine too
[15:04:48] <jynus>	 no lag, no ongoing things
[15:05:00] <jynus>	 I will change now dbstore1001
[15:05:12] <jynus>	 but that should not affect you
[15:08:09] <marostegui>	 ok
[15:08:51] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3210094 (10Papaul) mw2017 plug into ps1-a3 has jut one PSU and the reading on pss1-a3 is higher than ps2-a3 I will like to power that server dow...
[15:08:55] <jynus>	 after that, I may start with some of the changes you have not started yet
[15:09:31] <marostegui>	 sure
[15:09:39] <marostegui>	 going with s7 now
[15:10:10] <jynus>	 thanks
[15:27:04] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3210176 (10Papaul) The reading on both PUD's shows  ps1-a3 X=9.73 Y= 9.65 Z=12.84 ps2-a3  X=1.96 Y=9.65 z= 1.97  ps1-a3 is pulling more power th...
[15:28:05] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3210178 (10RobH) We should be able to balance it while the other tower 2 isnt being used, it will be more difficult but should be possible.
[15:42:11] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3210283 (10Marostegui) s7 is done. db1094 needs the table reimported
[15:44:30] <jynus>	 this is interesting
[15:44:43] <jynus>	 I cannot point dbstore1001 to db1062
[15:44:52] <marostegui>	 why??
[15:44:56] <jynus>	 because I have no logs from yesterday
[15:45:01] <marostegui>	 oh
[15:45:13] <jynus>	 so I have to wait until tomorroqw
[15:45:20] <marostegui>	 haha yeah
[15:45:24] <jynus>	 we actually did quite a risky movement
[15:45:33] <jynus>	 by pooling as master a recently created node
[15:45:49] <jynus>	 but it is what it is :-)
[15:45:56] <marostegui>	 we still have the logs from the other host, but it can be a pain if we need to recover stuff yes
[15:46:01] <jynus>	 if we are in an emergency
[15:46:10] <jynus>	 I could roll forward dbstore1001
[15:46:22] <jynus>	 but we are not, so I will do that tomorrow, too
[15:46:30] <marostegui>	 ok :)
[15:46:43] <marostegui>	 thanks for all the switchovers today, it has been amazing
[15:47:03] <jynus>	 not really, that should be a script
[15:47:12] <jynus>	 the script would have failed anyway
[15:47:17] <jynus>	 but yeah
[15:47:40] <jynus>	 maybe that is why change master failed?
[15:49:08] <marostegui>	 but the other host that failed should have had logs, no?
[15:49:14] <marostegui>	 or maybe not...
[15:49:17] <marostegui>	 it was new too
[15:49:51] <jynus>	 I am starting to get confused with how it actually works when going to the details
[15:50:20] <jynus>	 and I think it is starting to add lots of garbage to the status
[15:52:38] <wikibugs__>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3210346 (10jcrespo) Pending s5 eqiad master and dbstore1001 master change for s7.
[15:54:18] <jynus>	 should we do T153300 or should we just decomm it?
[15:54:18] <stashbot>	 T153300: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300
[15:54:29] <marostegui>	 nah, we can forget about it
[15:54:33] <marostegui>	 it is going to be decommed
[15:54:43] <marostegui>	 I would fix it maybe if we want to use it to reclone
[15:54:58] <jynus>	 that is actually a good reason
[15:55:12] <marostegui>	 let's keep it and we can alter it later
[15:55:18] <jynus>	 lower priority now
[15:55:21] <jynus>	 yeah
[15:55:22] <marostegui>	 yup
[15:56:07] <marostegui>	 check your email
[15:56:26] <marostegui>	 That is the summary of today 
[15:56:39] <marostegui>	 Pretty intense day!
[15:57:31] <jynus>	 any suggestion that you have not started with yet?
[15:57:44] <jynus>	 the primary keys one?
[15:57:54] <jynus>	 I will start with that
[16:00:30] <marostegui>	 tag_summary and change_tag?
[16:00:38] <marostegui>	 feel free yes
[16:01:12] <marostegui>	 i did only the test with etwiki
[16:01:19] <marostegui>	 but if it has not crashed yet, I guess it wont
[16:01:24] <jynus>	 no
[16:01:26] <jynus>	 the other tables
[16:01:30] <jynus>	 the ones on the list
[16:01:35] <marostegui>	 aaaah
[16:01:41] <marostegui>	 the ones from unique to pk?
[16:01:46] <jynus>	 yes
[16:01:52] <marostegui>	 yes yes yes yes yes yes!
[16:01:55] <marostegui>	 do those :)
[16:02:07] <jynus>	 I will research the templatelinks pagelinks and other errors
[16:02:13] <marostegui>	 great thank you
[16:02:51] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3210367 (10Papaul) I moved the  PSU's that are pulling lest power  too ps1-a3 and the once pulling more power on ps2-a3 we should be good for no...
[16:12:09] <wikibugs>	 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3210378 (10RobH) @Papaul: That will need to be swapped back once we fix the bios on the machines.  Idelally all PSU1 are pulling from PDU1(tower...
[16:14:44] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3210387 (10Marostegui) >>! In T130067#3210283, @Marostegui wrote: > s7 is done. db1094 needs the table reimported...
[16:14:54] <marostegui>	 I am going to logoff now, going to meet Arzhel for a drink in a bit
[16:15:04] <jynus>	 cheers
[16:15:07] <marostegui>	 Thanks for all the great work today Jaime, pretty productive day
[16:15:15] <jynus>	 no, thank YOU
[16:15:26] <jynus>	 I did nothing today, you did
[16:15:56] <marostegui>	 yeah right: https://phabricator.wikimedia.org/T162133 -> that is nothing, yes yes
[16:15:59] <marostegui>	 actually
[16:16:14] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3210399 (10Marostegui) a:03jcrespo
[16:16:16] <marostegui>	 ^ because you are doing it yourself!
[16:16:26] <marostegui>	 and now, I am off!! See you tomorrow :)
[16:16:39] <jynus>	 bye
[16:19:40] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3210420 (10jcrespo) As a note for @Addshore - the field should not be used or depend on it until tables are reimpor...
[16:20:38] <wikibugs__>	 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#2875886 (10jcrespo) db1041 is no longer a master- this can be done at any time now, and doesn't have to be done during the switchover.
[16:23:36] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3210425 (10Addshore) >>! In T130067#3210420, @jcrespo wrote: > As a note for @Addshore - the field should not be us...
[16:24:26] <addshore>	 Thanks for the works both! :)
[16:33:49] <wikibugs>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210490 (10jcrespo) a:03jcrespo So, for enwiki the updated list (having into account the tl and pl error...
[17:47:14] <wikibugs__>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210833 (10jcrespo) Results (including errors):  ```  root@db1075.eqiad.wmnet[(none)]> use etwiki Database...
[17:49:54] <wikibugs>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210840 (10jcrespo) No errors on labs or sanitariums.  Main concerns:  * Duplication of indexes on tl and...
[17:50:57] <wikibugs__>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210846 (10jcrespo) Above list minus the errors:  ``` alter table templatelinks add primary key (tl_from,t...
[18:17:19] <wikibugs>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210917 (10jcrespo) With this change + watchlist + tags, the list of tables without a primary key that are...
[18:19:48] <wikibugs>	 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3210922 (10jcrespo) Most are not too large, but oldimage will cause problem on commons in the future. We'l...