[06:44:17] <wikibugs>	 10Blocked-on-schema-change, 10GlobalUsage, 10Schema-change, 10User-DannyS712: GlobalUsage table `globalimagelinks` lacks a primary key - https://phabricator.wikimedia.org/T243987 (10Marostegui) confirmed that the PK exists on all s4 hosts for `commonswiki`. confirmed that the PK doesn't exist on any s4 hos...
[06:51:52] <wikibugs>	 10Blocked-on-schema-change, 10GlobalUsage, 10Schema-change, 10User-DannyS712: GlobalUsage table `globalimagelinks` lacks a primary key - https://phabricator.wikimedia.org/T243987 (10Marostegui) 05Open→03Resolved Change deployed on s4 primary master for `testcommonswiki`: ` labsdb1012.eqiad.wmnet:3306...
[06:51:54] <wikibugs>	 10DBA, 10MediaWiki-General, 10PostgreSQL, 10Schema-change, 10Wikimedia-database-error: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441 (10Marostegui)
[09:26:29] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10Marostegui) 05Open→03Resolved This was done. MySQL downtime was 60 seconds: ` Starts: 9:00:29 End: 9:01:29 ` Thanks so much everyone wh...
[09:26:33] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[09:26:54] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[09:38:01] <jynus>	 Re: labsdb1011 (I think it is that one) we can test to restart mysql and or the host when under low load
[09:38:16] <jynus>	 I am starting to see a patern of hosts randomly getting slower, potentially
[09:38:38] <jynus>	 if it is compression or something else, cannot say
[09:38:42] <marostegui>	 but labsdb1011 got rebuilt a few weeks ago, and it was actually showing very good performance
[09:38:51] <jynus>	 I may be wrong, then
[09:38:51] <marostegui>	 at least replication-thread wise
[09:38:58] <marostegui>	 no, it could totally be
[09:38:59] <marostegui>	 we'll see
[09:39:04] <jynus>	 I haven't dig too far into the tickets
[11:46:53] <jynus>	 Re: connections, s7 seems to have become a bit spikier lately: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&from=1584576769663&to=1584618346429&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13317&fullscreen&panelId=5
[11:47:15] <jynus>	 (maybe it is just dumps or something else)
[12:44:22] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui)
[12:44:32] <marostegui>	 Amir1: I have created this ^ instead of using the epic just for DBAs
[12:45:11] <Amir1>	 marostegui: sure
[12:45:52] <marostegui>	 Amir1: I assume the only table that we have to keep in labs for now is wikidatawiki.wb_terms or do we have to also keep s4 (commons, testcommons - they are empty), and s3 (testwikidatawiki - which has around 500k rows)
[12:45:53] <Amir1>	 Thanks
[12:46:11] <Amir1>	 marostegui: only wikidatawiki
[12:46:25] <marostegui>	 cool
[12:46:27] <marostegui>	 I will note that
[12:47:12] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui)
[12:47:29] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) a:03Marostegui
[12:57:47] <marostegui>	 Amir1: Should we go ahead and rename it on testwikidatawiki on s3 eqiad?
[12:58:20] <marostegui>	 And if all goes fine today and tomorrow, on monday we can go ahead and rename it on s8 eqiad
[12:58:23] <marostegui>	 on a slave there
[12:58:30] <Amir1>	 marostegui: sure but let's double check things first, is the replication working? is the files stayed untouched?
[12:58:44] <Amir1>	 also can you grep binlogs for wb_terms?
[12:58:48] <marostegui>	 Replication is working yeah, and no writes happened, otherwise replication would have broken
[12:59:12] <marostegui>	 -rw-rw---- 1 mysql mysql 529G Mar 18 12:19 wb_terms.ibd
[12:59:13] <marostegui>	 ^ s8
[12:59:19] <marostegui>	 same time as yesterday
[12:59:26] <Amir1>	 reads are more important. Can we check it somehow?
[12:59:48] <marostegui>	 -rw-rw---- 1 mysql mysql 312M Mar 17 07:22 wb_terms.ibd
[12:59:50] <marostegui>	 ^ s3
[13:00:06] <marostegui>	 Amir1: we cannot check reads :_(
[13:00:42] <marostegui>	 let me see if I can get someting out of sys or performance_schema
[13:00:50] <Amir1>	 thanks
[13:03:16] <marostegui>	 Amir1: nah, I don't think we can do anything without enabling the query log
[13:04:15] <marostegui>	 I can enable it for a few minutes and try to grep there, but other than that I cannot do much more than that
[13:05:40] <Amir1>	 if there are reads, they might happen once in a few hours by some fringe code paths in some weird extension (I checked everything but you know)
[13:05:59] <Amir1>	 renaming it in a several replica would do I guess
[13:06:10] <Amir1>	 we can do it later in the next week
[13:07:46] <marostegui>	 Amir1: I wanted to go only for 1 replica on s3
[13:07:49] <marostegui>	 not s8
[13:08:22] <Amir1>	 yeah, I thought it would have been the last resort 
[13:08:33] <Amir1>	 but since there's none, let's do it
[13:09:20] <marostegui>	 So whatever that reads from s3, will fail if it hits that particular replica
[13:09:52] <Amir1>	 yeah, that sounds pretty good
[13:10:08] <marostegui>	 ok, let me check which one
[13:11:18] <marostegui>	 db1078 will be
[13:12:25] <Amir1>	 great
[13:12:44] <marostegui>	 done
[13:12:44] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) Table renamed on s3 (testwikidatawiki): ` root@db1078.eqiad.wmnet[testwikidatawiki]> set session sql_log_bin=0; rename table wb_terms to T...
[13:12:49] <marostegui>	 now monitoring logstash for it
[13:19:33] <marostegui>	 Amir1: what should we do with the table on the analytics dbstore? 
[13:19:41] <marostegui>	 That is dbstore100X..
[13:19:45] <marostegui>	 Should we ask Luca?
[13:20:44] <Amir1>	 marostegui: yeah, good question, let me ask PM/EM
[13:20:57] <marostegui>	 We can ping luca in sre
[13:22:19] <marostegui>	 I guess testwikidatawiki is out of the question here, it is just s8
[13:23:07] <Amir1>	 yeah
[13:23:21] <Amir1>	 I can send an email to research-internal
[13:23:37] <Amir1>	 asking if anyone is using the table and if they need more time
[13:26:16] <marostegui>	 cool, can you CC Me?
[13:26:42] <jynus>	 SELECT rows_fetched FROM schema_table_statistics WHERE table_name='wb_terms'; will tell you if the table is being read
[13:27:14] <Amir1>	 sure
[13:28:30] <jynus>	 marostegui: https://phabricator.wikimedia.org/P10729
[13:29:13] <marostegui>	 Oh, I didn't know that one!
[13:29:20] <marostegui>	 Thanks!
[13:30:18] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui)
[13:31:16] <Amir1>	 niiice
[13:31:38] <wikibugs>	 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) Current stats on db1126: ` root@db1126.eqiad.wmnet[sys]> SELECT rows_fetched FROM schema_table_statistics WHERE table_name='wb_terms'; +--...
[13:31:41] <marostegui>	 ^ added to the task!
[13:31:52] <marostegui>	 for db1126
[13:32:01] <jynus>	 important reads, of course, would be on all of them :-D
[13:32:21] <jynus>	 also have: SELECT * FROM io_global_by_file_by_bytes where file='@@datadir/wikidatawiki/wb_terms.ibd';
[13:32:34] <jynus>	 but that is file stats which are not 1 to 1 to table stats
[13:32:53] <marostegui>	 yeah, I did know that one
[13:33:27] <marostegui>	 but the other one is a really good one!
[13:35:57] <marostegui>	 I have added all the stats, just in case someone reads that table for whatever reason, manually I mean
[13:43:50] <jynus>	 thanks
[13:47:14] <marostegui>	 thanks for the email Amir1 
[13:47:23] <Amir1>	 marostegui: oh don't mention it :)
[13:48:01] <jynus>	 would tool users need a reminder of changes?
[13:49:23] <marostegui>	 jynus: I think it was asked yesterday or few days ago and Amir1 (or Adam) said they'd announce it again
[13:49:27] <jynus>	 maybe another email to Bryan re:[Cloud-announce] [Breaking change]
[13:49:40] <jynus>	 ok, so already taken into account
[13:49:46] <marostegui>	 yep I believe so
[13:50:04] <jynus>	 the email is accurate
[13:50:09] <Amir1>	 Yeah. That's on the radar
[13:50:12] <jynus>	 "will be dropped in may"
[13:50:19] <jynus>	 didn't specify the year!
[13:50:23] <Amir1>	 Yup
[13:50:49] <Amir1>	 It was May 2019 but for the reasons I told you in FOSEDM it had be delayed
[13:50:59] <jynus>	 oh, not worried at all
[13:51:15] <jynus>	 if something, you know I ve been pushing for doing it slower
[13:55:19] <Amir1>	 no, the thing is that it was postponed for year, it took engineering time and resources for 8 more months than expected 
[13:55:33] <Amir1>	 it was planned for three, took 11
[15:03:31] <wikibugs>	 10DBA, 10Upstream: Possibly disable optimizer flag: rowid_filter on 10.4 - https://phabricator.wikimedia.org/T245489 (10Marostegui) I asked MariaDB and looks like they think they'll be able to include this fix on the next 10.4.13 that is scheduled to be shipped by the end of April