[05:58:39] <wikibugs>	 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 05WMF-NDA: Drop tables with no events in last 90 days. - https://phabricator.wikimedia.org/T161855#3145706 (10Marostegui) That is fine, thanks for the heads up. Normally I like to run those statements with:  ``` set session sql_log_bin=0 ; DROP TABLE IF...
[07:41:50] <wikibugs_>	 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#3145821 (10Marostegui) Will leave this alter table running on db2019 (codfw master) on Monday, it will generate lag on codfw.
[07:51:26] <wikibugs_>	 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 05WMF-NDA: Drop tables with no events in last 90 days. - https://phabricator.wikimedia.org/T161855#3145859 (10jcrespo) +1
[08:20:53] <wikibugs>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3145898 (10Marostegui) Things that we need to do while codfw is active: - ALTER eqiad s4 master: T73563 - ALTER eqiad shards: T130067 - ALTER eqiad shards: T147166 - ALTER eqiad masters (if we finish on time the sl...
[08:21:14] <wikibugs_>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3145903 (10Marostegui)
[08:35:08] <jynus>	 marostegui, do you remember which ticket did I transform the special slaves on?
[08:35:21] <jynus>	 it was a bug report more or less handled by anomie
[08:35:30] <marostegui>	 i think so yes
[08:35:32] <marostegui>	 I know the key word
[08:35:38] <jynus>	 what was it
[08:35:39] <marostegui>	 (if it is the one I believe)
[08:35:40] <marostegui>	 give me a sec
[08:35:51] <marostegui>	 https://phabricator.wikimedia.org/T147747
[08:35:53] <marostegui>	 is that one?
[08:36:09] <jynus>	 yes, thank you
[08:36:12] <marostegui>	 yw!
[08:36:24] <jynus>	 I want to see why the special slave is not special
[08:36:32] <jynus>	 if it was moved, maintenance, etc.
[08:38:44] <jynus>	 how could I miss an unpartitioned slave? maybe it didn't have the role at the time?
[08:40:12] <marostegui>	 look at this
[08:40:13] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/297235/
[08:41:08] <jynus>	 but it doesn't say why
[08:41:25] <marostegui>	 failover maybe db1034 failed?
[08:41:55] <jynus>	 db1034 seem to have the right indexes
[08:42:26] <jynus>	 I am going to use it back and see how it evolves
[08:42:46] <jynus>	 db1062 will need to be depooled for alter anyway
[08:42:59] <marostegui>	 there is no ticket related to db1034 around that date (or I cannot find it)
[08:50:59] <wikibugs_>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3145999 (10jcrespo) > Feel free to edit this comment and add/change stuff as needed.  Sadly, phabricator is not a wiki- either put it on the header, a wiki or an etherpad :-)
[08:51:47] <jynus>	 ^I cannot edit your comment
[08:51:55] <marostegui>	 ah! no worries, will do it now :)
[08:52:12] <marostegui>	 right you can only edit the task description :)
[08:55:49] <wikibugs>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146014 (10Marostegui) https://etherpad.wikimedia.org/p/dba-dc-tasks
[08:56:03] <wikibugs_>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146015 (10Marostegui)
[09:01:52] <jynus>	 I think I am going to move it to a wiki- I am adding the commands and I do not want to non notice changes there
[09:01:58] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3146023 (10thiemowmde)
[09:02:28] <marostegui>	 sure, that is fine too
[09:02:39] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3137680 (10thiemowmde) p:05High>03Low
[09:02:57] <jynus>	 imagine somone makes a joke and puts a DROP there
[09:03:10] <marostegui>	 yeah, that's true XD
[09:11:27] <wikibugs>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146044 (10jcrespo) I have migrated that to the wiki for security reasons, as I have added details of that and added some extra tasks , and questions about others: https://wikitech.wikimedia.org/wiki/Switch_Datacen...
[09:11:36] <jynus>	 https://wikitech.wikimedia.org/wiki/Switch_Datacenter/planned_db_maintenance
[09:11:47] <jynus>	 We should put them in order of importance
[09:12:41] <jynus>	 try to do as many as possible, but stop at some point if we run out of time
[09:13:28] <marostegui>	 agreed
[09:13:37] <jynus>	 I have done some edits
[09:13:42] <marostegui>	 me too
[09:13:43] <jynus>	 check it when you have time
[09:14:42] <jynus>	 the master failover I have not told you
[09:14:52] <marostegui>	 no, you haven't
[09:14:52] <jynus>	 thinking about decomissioning old servers
[09:14:56] <marostegui>	 ah
[09:15:05] <marostegui>	 yes, we have some masters that will go away
[09:15:07] <jynus>	 we probably want to minimize distruption
[09:15:07] <marostegui>	 good catch
[09:15:09] <jynus>	 if possible
[09:15:24] <marostegui>	 yes, it is a good idea
[09:15:26] <jynus>	 by putting newer masters
[09:15:35] <jynus>	 needs work
[09:15:59] <marostegui>	 yes
[09:28:38] <wikibugs>	 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146087 (10jcrespo)
[09:40:17] <jynus>	 db1066.eqiad.wmnet has issues with the new code
[09:40:33] <marostegui>	 errors?
[09:40:34] <jynus>	 https://logstash.wikimedia.org/goto/de5e7f665991fc8c771874fac75098ac
[09:40:51] <marostegui>	 of :(
[09:41:05] <jynus>	 not sure why it is executed there
[09:41:15] <jynus>	 but I am going to depool it
[09:41:42] <marostegui>	 mmm it has the correct PK, but not the index
[09:42:41] <marostegui>	 you want me to add it?
[09:43:12] <jynus>	 first:
[09:43:21] <jynus>	 https://gerrit.wikimedia.org/r/345811
[09:43:27] <jynus>	 then we can discuss :-)
[09:43:45] <marostegui>	 +1ed
[09:44:09] <jynus>	 ah, I see
[09:44:21] <jynus>	 unlike the other fix
[09:44:30] <jynus>	 this wasn't pinned to a single role
[09:44:38] <marostegui>	 the other API servers do have the index
[09:44:46] <jynus>	 yeah, it has to be only that
[09:44:56] <marostegui>	 yeah it is weird because it has the correct PK but not the indexes
[09:45:00] <jynus>	 because I only see this host with errors in the last hours
[09:45:03] <marostegui>	 whereas the other have the whole thing "wrong"
[09:45:08] <marostegui>	 pk, and all that
[09:45:13] <marostegui>	 once it is depooled I can run the alter
[09:45:15] <marostegui>	 I have it handy
[09:45:17] <jynus>	 we stopped enwiki alter
[09:45:27] <jynus>	 that was probably going to happen
[09:45:43] <jynus>	 now we need the alters, rathern than being blocked :-)
[09:46:45] <jynus>	 we need the index to exist so we can ignore it
[09:46:50] <marostegui>	 XDDD
[09:47:03] <marostegui>	 ok, i will deploy it once you've depooled it
[09:47:12] <jynus>	 probably index hints should be done so that they only work if indexes exist
[09:47:32] <jynus>	 but that is difficult to do with performance
[09:47:43] <jynus>	 whithout doubling the roundtrip
[09:47:58] <jynus>	 or keeping expensive cache over 40000 dbs
[09:51:01] <marostegui>	 i see you deployed already
[09:51:07] <marostegui>	 want me to run the alter then?
[09:51:25] <jynus>	 yeah, no more errors now
[09:51:28] <marostegui>	 ok
[09:52:02] <jynus>	 that was probably the only server without that index in the whole cluster
[09:52:18] <marostegui>	 running now
[09:53:37] <jynus>	 isn't T159319 fun?
[09:54:53] <jynus>	 if it lacks other indexes, we can add them now "for free", too :-)
[09:56:37] <marostegui>	 haha
[09:57:11] <jynus>	 we have a 2x1 offer in indexes today
[09:58:21] <marostegui>	 special Friday deal?
[10:00:03] <jynus>	 the other "doesn't exist in table" in the last 7 days is 2 requests to arbcom_arwiki, which I fixed previously this week- and those were probably tests, not real traffic
[10:01:04] <marostegui>	 good good
[10:01:19] <marostegui>	 let's see how long this one takes
[10:03:56] <jynus>	 for the other change, eswiki seems happy
[10:04:02] <marostegui>	 nice
[10:04:23] <jynus>	 (I only notice that issue when I was about to check the other deploy)
[10:05:12] <marostegui>	 yeah, it is kind of a constant stream of an average of 80 errors or so in logtash
[10:05:17] <marostegui>	 so not that that that much
[10:05:23] <marostegui>	 but not nice indeed to leave it like that the weekend
[10:05:45] <jynus>	 the problem is the noise
[10:05:53] <jynus>	 so many closure errors
[10:05:57] <jynus>	 so many long running queries
[10:06:09] <jynus>	 that when a real issue happens- we do not notice
[10:09:58] <wikibugs_>	 10DBA, 07Epic, 05codfw-rollout: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146214 (10Aklapper)
[11:15:32] <wikibugs_>	 10DBA, 13Patch-For-Review: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) - https://phabricator.wikimedia.org/T154485#3146307 (10jcrespo) There are a lot of differences between the master and dbstore1002 on:  ``` itwiki plwiki svwiki ```  One (maybe...
[11:18:49] <wikibugs_>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3146329 (10Marostegui) db1066 is done: ``` root@neodymium:/home/marostegui# mysql --skip-ssl -hdb1066 enwiki -e "show create table revision\G" *************************...
[11:24:31] <jynus>	 error count seems to be low again
[11:24:48] <marostegui>	 :)
[11:25:07] <jynus>	 I would leave that ticket open for a week
[11:25:11] <marostegui>	 yes
[11:25:12] <marostegui>	 agreed
[11:25:21] <marostegui>	 and monitor every morning
[11:25:27] <marostegui>	 to catch it if it happens
[12:35:58] <jynus>	 Based on the latest issues, I have slightly modified the kibana DBQuery Dashboard
[12:35:58] <jynus>	 hopefully for the better
[12:37:03] <jynus>	 feedback is welcome
[13:17:02] <paravoid>	 modules/mariadb/manifests/packages.pp seems to be entirely precise-based; is this needed?
[13:17:21] <paravoid>	 same for modules/mysql/manifests/server/package.pp
[13:25:19] <jynus>	 paravoid I will have a look at it- we do not use package.pp at all on production
[13:25:28] <jynus>	 we use package_wmf.pp
[13:25:35] <paravoid>	 yeah, it looks like it since it relies on some precise packages :)
[13:25:42] <paravoid>	 so let's kill the dead code?
[13:25:51] <jynus>	 maybe it is time to delete it and rename package_wmf -> package
[13:25:57] <paravoid>	 nod
[13:26:02] <jynus>	 I am worried if someone uses the other
[13:26:09] <jynus>	 let me check it properly
[13:26:13] <paravoid>	 noone should be using precise
[13:26:24] <jynus>	 yeah, I understand
[13:26:32] <jynus>	 but I prefer to kill it fully
[13:26:36] <jynus>	 if possible
[13:26:42] <paravoid>	 sure, by all means :)
[13:26:45] <jynus>	 let me ask if someone uses the non _wmf
[13:26:52] <jynus>	 and I will either fix it or delete it
[13:27:04] <paravoid>	 like who?
[13:27:11] <paravoid>	 other users of our puppet repository you mean?
[13:27:15] <jynus>	 yes
[13:27:24] <jynus>	 labs/beta/etc
[13:27:30] <paravoid>	 labs doesn't have precise anymore either
[13:27:38] <jynus>	 this is not about procise
[13:27:46] <jynus>	 this is about mariadb-server
[13:27:51] <jynus>	 vs wmf-mariadb10
[13:27:59] <paravoid>	 sort of
[13:28:12] <paravoid>	 it relies on packages from the precise-wikimedia repository
[13:28:18] <paravoid>	 which will very soon cease to exist
[13:28:22] <jynus>	 I am not going to fix it if nobody uses it
[13:28:28] <jynus>	 I prefer to just delete it
[13:28:35] <paravoid>	 and presumably packages in the precise-wikimedia repository are built against precise libraries etc. as well
[13:28:37] <jynus>	 let me ask around
[13:28:42] <jynus>	 and I will fix it
[13:28:44] <paravoid>	 ok
[13:28:44] <jynus>	 do not worry
[13:29:25] <jynus>	 even if someone uses it, I think we can merge both files os they are properly mantained
[13:29:46] <jynus>	 unlike now, where I do not touch the packages.pp
[13:30:12] <paravoid>	 I intend to kill the precise-wikimedia suite in our apt repository on Monday
[13:30:31] <paravoid>	 so the packages won't be there anymore
[14:02:15] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3146611 (10jcrespo) a:03jcrespo I am kickstarting the replication right now- this required different pupetization of the repliation "grants". I wi...
[14:02:34] <marostegui>	 \o/
[14:06:15] <jynus>	 paravoid, I think I wasn't clear enough before. My fault.
[14:06:16] <jynus>	 I think there could be trusty instances using that class
[14:06:38] <jynus>	 (yes, even with the precise stuff)
[14:06:47] <paravoid>	 ok
[14:06:48] <jynus>	 I want to delete directly
[14:07:00] <jynus>	 rather than just the first patch I sent
[14:07:03] <jynus>	 that can go now
[14:07:32] <jynus>	 I will work to delete now the whole class fully- and that will take more (becase it is not on service I own)
[14:07:44] <jynus>	 services*
[14:26:31] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3146645 (10matej_suchanek)
[14:30:20] <wikibugs_>	 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3146654 (10Marostegui) Hi! How's the process to decommission db1047 going?
[15:26:50] <wikibugs>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#3146758 (10jcrespo) 05Open>03Resolved a:03jcrespo This is done. Some small follups (not related to jessie) at: T157359
[15:54:49] <wikibugs_>	 10DBA, 07Epic, 05codfw-rollout: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3146861 (10Marostegui)
[16:17:24] <wikibugs_>	 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3146925 (10jcrespo) Ok, now puppet works, but either it puppet needs more work or it fails silently-this needs more researech. Replication is not wo...
[17:06:07] <wikibugs>	 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#3147093 (10Ottomata) ping @Marostegui, in case you didn't see it: https://gerrit.wikimedia.org/r/345646