[05:20:49] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4181114 (10Marostegui)
[05:21:09] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4181115 (10Marostegui)
[05:21:11] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4181116 (10Marostegui)
[05:25:19] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T193747#4181121 (10Marostegui) 05Open>03Resolved All fixed! ``` root@db1063:~# megacli -LDInfo -L0 -a0   Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name                : RAID Level...
[05:26:43] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4181126 (10Marostegui) s4 eqiad progress  [] db1102 [] labsdb1009 [] labsdb1010 [] labsdb1011 [] dbstore1...
[05:27:03] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4181127 (10Marostegui) s4 eqiad progress  [] db1102 [] labsdb1009 [] labsdb1010 [] labsdb1011 [] dbstore1002 []...
[05:27:07] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4181128 (10Marostegui) s4 eqiad progress  [] db1102 [] labsdb1009 [] labsdb1010 [] labsdb1011 []...
[05:27:34] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4181129 (10Marostegui)
[05:27:50] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4181130 (10Marostegui)
[05:28:06] <wikibugs_>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4181131 (10Marostegui)
[05:34:22] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4181132 (10Marostegui) 05Open>03Resolved Let's create a general task if this happens again and consider this fixed for now and for this host.
[05:42:21] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Move db1067 to row C - https://phabricator.wikimedia.org/T193835#4181136 (10Marostegui)
[05:43:00] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Move db1067 to row C - https://phabricator.wikimedia.org/T193835#4181148 (10Marostegui) p:05Triage>03Normal
[06:00:17] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4181155 (10Marostegui)
[08:26:41] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4181279 (10Marostegui)
[08:27:20] <wikibugs_>	 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4156169 (10Marostegui)
[08:29:38] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4181302 (10Marostegui) >>! In T193325#4177621, @jcrespo wrote: > The check detected some difference, but they could be false positives, checking again....
[08:29:59] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4181303 (10Marostegui) p:05High>03Normal
[09:03:42] <wikibugs_>	 10DBA, 10Cloud-Services, 10Patch-For-Review: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#4181364 (10jcrespo) @kaldari Counting is available to be done on production (or anywhere, throught the api). There is not such an infrastructure se...
[09:10:24] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4181389 (10jcrespo) Yes, no differences were assured on a second run. I will repool the server now.
[09:40:07] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Move db1066 to row A - https://phabricator.wikimedia.org/T193847#4181453 (10Marostegui)
[09:40:27] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: Move db1066 to row A - https://phabricator.wikimedia.org/T193847#4181465 (10Marostegui) p:05Triage>03Normal
[09:48:02] <wikibugs_>	 10DBA: Drop old devwikiinternal and rel13testwiki from s3 - https://phabricator.wikimedia.org/T118764#4181474 (10Marostegui) @Reedy @MaxSem you guys fine if I go ahead and drop these two?
[10:47:36] <wikibugs_>	 10DBA, 10Datasets-General-or-Unknown, 10Dumps-Generation, 10MW-1.28-release (WMF-deploy-2016-06-21_(1.28.0-wmf.7)), and 3 others: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#4181637 (10ArielGlenn) I need to go through the logs fo...
[10:57:21] <wikibugs_>	 10DBA, 10Patch-For-Review: Set barracuda InnoDB file format as the default configuration everywhere - https://phabricator.wikimedia.org/T150949#4181677 (10Marostegui) p:05Triage>03Normal a:03Marostegui
[11:16:30] <jynus>	 I am going to stop db1056 now, schedule for decom next week
[11:17:01] <marostegui>	 cool
[11:30:49] <jynus>	 probably you do not want to migrate to db1116 as it is not the final hardware
[11:31:27] <jynus>	 however, given the timeframe we are moving, it would be advantageos to start using it to 1) either free db1095 for other uses
[11:31:40] <jynus>	 or at least 2) avoid unnecesary shared lag
[11:31:51] <jynus>	 (doesn't have to happen today, though)
[11:36:48] <wikibugs_>	 10DBA, 10Datasets-General-or-Unknown, 10Dumps-Generation, 10MW-1.28-release (WMF-deploy-2016-06-21_(1.28.0-wmf.7)), and 3 others: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#4181784 (10ArielGlenn) List of wikis with more than 10...
[11:39:49] <wikibugs_>	 10DBA: Decommission db1056 - https://phabricator.wikimedia.org/T193736#4181805 (10jcrespo)
[11:40:07] <wikibugs_>	 10DBA: Decommission db1056 - https://phabricator.wikimedia.org/T193736#4178332 (10jcrespo) We will wait until Monday and then send it to dc-ops.
[11:54:29] <wikibugs_>	 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4181845 (10jcrespo)
[12:25:21] <jynus>	 if you are around, Amir1, we can check your patch after my lunch, I was on purpose delaying it because we got load issues on s8 on dbstore2001
[12:25:31] <jynus>	 but it may be all right
[12:44:24] <wikibugs_>	 10DBA, 10Datasets-General-or-Unknown, 10Dumps-Generation, 10MW-1.28-release (WMF-deploy-2016-06-21_(1.28.0-wmf.7)), and 3 others: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#4181972 (10ArielGlenn) Mae a custom dblist file with th...
[12:56:03] <Amir1>	 jynus: good, I'm around
[12:56:05] <Amir1>	 let me know
[12:58:02] <wikibugs_>	 10DBA: Drop old devwikiinternal and rel13testwiki from s3 - https://phabricator.wikimedia.org/T118764#4181990 (10Reedy) I'd say so. You've got a backup of the tiny amount of data they apparently hold... So if anyone needs them, we have it :)
[13:49:29] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4182219 (10jcrespo) 05Open>03Resolved
[14:15:50] <wikibugs_>	 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#4182285 (10Marostegui)
[14:15:54] <wikibugs_>	 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4182284 (10Marostegui) 05Open>03stalled
[14:21:09] <jynus>	 Amir1: I cannot find the patch
[14:21:17] <wikibugs_>	 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591#4182293 (10Marostegui) This one should be an easy one as the table is empty everywhere, so it could be even done with replication directly on the masters
[14:54:07] <Amir1>	 jynus: https://gerrit.wikimedia.org/r/#/c/427202/
[14:54:14] <Amir1>	 (sorry I was afk for lunch/dinner)
[14:55:29] <jynus>	 I just found it
[14:56:23] <jynus>	 should I move the logs to the path given?
[15:01:17] <Amir1>	 jynus: yes, otherwise it doesn't work
[15:03:41] <jynus>	 let me manually change the cron time and you can check the logs/actions?
[15:04:31] <Amir1>	 sure
[15:05:09] <jynus>	 it should have started now
[15:05:34] <Amir1>	 jynus: btw. I'm taking over this https://phabricator.wikimedia.org/T185355 it needs creating tables and alter tables 
[15:06:10] <jynus>	 daniel's advice is quite wise
[15:07:05] <jynus>	 we may start having some lag on dbstore2001
[15:07:21] <jynus>	 but the others are in the clear
[15:08:02] <jynus>	 let me kill the run, and run puppet so we do not accidentally run 2 instances of this
[15:08:41] <jynus>	 and so we have 20 minutes to validate the changes
[15:09:38] <jynus>	 i've killed it
[15:10:03] <jynus>	 I wonder if the report is too frequent
[15:10:08] <Amir1>	 jynus: It's fine
[15:10:33] <jynus>	 it is ok, only 78 entries in 5 minutes
[15:10:44] <jynus>	 it seemed faster on command line
[15:11:10] <jynus>	 let me validate on the binlog so you are ok with the actions taken
[15:15:14] <jynus>	 UPDATE /* Wikibase\Repo\Store\Sql\TermSqlIndexSearchFieldsClearer::clearBatch www-data@terbiu... */  `wb_terms` SET term_search_key = '' WHERE term_row_id IN
[15:18:25] <jynus>	 Amir1: I've added it to https://wikitech.wikimedia.org/wiki/Deployments#Week_of_April_30th please copy it to the next weeks if it keeps going
[15:18:48] <Amir1>	 sure
[15:18:51] <Amir1>	 Thanks
[16:36:41] <musikanimal>	 hello DBAs and friends. Is there a formal process for requesting results of a production query? (for someone else, not me)
[16:36:54] <musikanimal>	 Specifically, this is to find AbuseFilter filters where the pattern includes a certain string, for development purposes
[16:36:57] <jynus>	 "results"?
[16:37:02] <jynus>	 is it private data?
[16:37:13] <jynus>	 ah
[16:37:26] <jynus>	 well, any deployer should have access to it
[16:37:35] <musikanimal>	 it is not replicated, no, because the filter pattern as a whole is "private" (chiefly just for counter-vandalism, not privacy related)
[16:37:42] <jynus>	 I see
[16:37:55] <jynus>	 so private for anti-abuse, not for privacy
[16:38:01] <musikanimal>	 correct
[16:38:22] <musikanimal>	 here we just want the filter ID, so the pattern doesn't even need to be shared
[16:38:26] <jynus>	 just put the query somewhere and people with access
[16:38:42] <jynus>	 like me
[16:38:50] <musikanimal>	 right, I can run the queries myself, I was just wondering if there was a formal process for this
[16:38:53] <jynus>	 but anyone with deployment access could do that
[16:39:14] <jynus>	 create a ticket just for tracking purposes
[16:39:28] <musikanimal>	 tag with data-services I guess?
[16:39:31] <jynus>	 I suppose it is part of an ongoing issues
[16:39:33] <jynus>	 nope
[16:39:47] <jynus>	 that is a cloud tag
[16:40:13] <jynus>	 DBA is ok, I guess
[16:40:34] <musikanimal>	 alright as long as you say it's OK :)
[16:40:59] <jynus>	 I think Wikimedia-databases is an alias of DBA
[16:41:06] <musikanimal>	 I know y'all are busy, and there are a lot of non-DBAs with prod db access
[16:41:35] <jynus>	 it is ok, just say anyone can do it on the description and it will be understood
[16:41:46] <jynus>	 site-requests maybe
[16:41:49] <musikanimal>	 got it. Thanks!
[16:42:12] <jynus>	 "Requests for MediaWiki system administration tasks to be performed on Wikimedia Foundation hosted wikis"
[16:42:18] <jynus>	 that would be ok too
[16:42:36] <jynus>	 we do not have a specific tag for "please run a query on production"
[16:43:10] <musikanimal>	 yeah, that's what I was wondering. I feel like this can't be uncommon
[16:43:44] <jynus>	 in most cases people use beta for this things
[16:44:11] <jynus>	 and only for production debug that is done, which should be exactly the same as beta
[16:44:36] <musikanimal>	 yeah, here this developer wants to know what filters will be affected by a change to the extension, that sort of thing
[16:45:02] <jynus>	 that is ok
[16:45:18] <jynus>	 just run the query and use phabricator to track it
[16:45:40] <musikanimal>	 okee doke :)
[16:46:14] <jynus>	 it is more of a "if it is common, maybe there is a need for access/lack of automation"
[17:26:55] <wikibugs_>	 10DBA, 10AbuseFilter: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4182744 (10Daimona)
[17:28:15] <wikibugs_>	 10DBA, 10AbuseFilter, 10Wikimedia-Site-requests: Run listed queries for all wikis in order to unblock existing changes - https://phabricator.wikimedia.org/T193894#4182759 (10Daimona)
[17:34:42] <wikibugs_>	 10DBA: Drop old devwikiinternal and rel13testwiki from s3 - https://phabricator.wikimedia.org/T118764#4182794 (10MaxSem) Yeah, no way anybody needs these.
[17:42:24] <wikibugs_>	 10DBA, 10Cloud-Services, 10Patch-For-Review: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#4182820 (10kaldari) @jcrespo: Thanks for the info. This tool will be running on Tool Forge. It may be possible for us to use the API for this, alth...