[04:17:44] <wikibugs>	 10DBA, 10Performance-Team, 10MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)), 10Patch-For-Review, 10Wikimedia-log-errors: Frequent "Wikimedia\\Rdbms\\DatabaseMysqlBase::lock failed to acquire lock" errors on WMF mediawiki logs - https://phabricator.wikimedia.org/T180793#3811463 (10aaron) By...
[04:17:56] <wikibugs>	 10DBA, 10Performance-Team, 10MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)), 10Patch-For-Review, 10Wikimedia-log-errors: Frequent "Wikimedia\\Rdbms\\DatabaseMysqlBase::lock failed to acquire lock" errors on WMF mediawiki logs - https://phabricator.wikimedia.org/T180793#3811464 (10aaron) 0...
[06:28:07] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) - https://phabricator.wikimedia.org/T162699#3811546 (10Marostegui)
[06:28:22] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) - https://phabricator.wikimedia.org/T162699#3171683 (10Marostegui)
[06:29:20] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3811555 (10Marostegui)
[06:29:22] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) - https://phabricator.wikimedia.org/T162699#3171683 (10Marostegui) 05Open>03Resolved All these servers have been decommissioned or have individual tasks for decommissioning as described above, so clo...
[08:32:18] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3811605 (10Marostegui)
[09:16:26] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3811698 (10Marostegui) db1098:3317 is now replicating
[09:21:34] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3811723 (10Marostegui) >>! In T178359#3811722, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/log/AWAl-nS2wg13V6286MaL} [2017-12-05T0...
[09:30:12] <wikibugs>	 10DBA, 10Community-Tech, 10MediaWiki-General-or-Unknown, 10Operations, and 3 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3811728 (10Marostegui) @EddieGP any update on this? is this resolved (as per your submitted patch?)
[09:55:02] <eddiegp>	 marostegui: About T176754, I'd actually love your (DBA) review of https://gerrit.wikimedia.org/r/#/c/384429/ . I think all we need for that patch to be merged is to be sure that your concerns (jynus had some comments on the first few patchsets) about it are resolved :)
[09:55:02] <stashbot>	 T176754: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754
[09:55:51] <eddiegp>	 After merging/deploying that one I'll poke someone to manually run that maintenance script. If that goes fine we can deploy the cron :)
[09:56:24] <marostegui>	 Ah right, I didn't notice it was not merged yet
[09:56:26] <marostegui>	 sorry :)
[09:57:51] <eddiegp>	 No problem :)
[10:01:40] <jynus>	 marostegui: dbstore1002 replication failed yesterday night again, I am making that a priority
[10:02:10] <marostegui>	 jynus: yeah, I fixed it this morning, same table. I didn't want to start the import as I didn't  know if you were already working on it or something. Just limited to fix it and get it running
[10:02:28] <jynus>	 wb_claims key, curious?
[10:02:33] <marostegui>	 yep
[10:02:36] <marostegui>	 that one
[10:02:42] <jynus>	 for some reason, that is missing everywhere
[10:02:57] <jynus>	 some delete or insert missing, or some period of time skipped
[10:03:00] <marostegui>	 that was also the one that gave trouble on sunday morning, for the record
[10:03:03] <jynus>	 yeah
[10:03:11] <marostegui>	 to you as well on saturday?
[10:03:14] <jynus>	 yeah
[10:03:26] <elukey>	 is there anything that I can do to help for dbstore1002?
[10:03:31] <jynus>	 I am reimporing the whole table, though
[10:03:43] <jynus>	 elukey: don't worry, nothing to do with analytics
[10:03:48] <jynus>	 this is pure data consistency
[10:05:06] <elukey>	 ok :)
[10:06:57] <jynus>	 I will stop dbstore1002 and dbstore2001 (both s5 only)
[10:09:14] <marostegui>	 col
[10:09:15] <marostegui>	 cool
[10:57:49] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3812016 (10jcrespo) @bd808 T174569#3810307 This is important for cloud.
[11:26:09] <jynus>	 dewiki page_props is reimported already, wikidatawiki will take over an hour
[13:07:53] <wikibugs>	 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10User-Daniel, 10Wikidata-Former-Sprint-Board: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3812451 (10Lucas_Werkmeister_WMDE)
[13:18:39] <jynus>	  wikidatawiki will probably take 3-4 hours
[13:18:49] <jynus>	 we are on the 2h still
[13:29:36] <marostegui>	 still worth the time of course!
[14:36:28] <jynus>	 almost there
[14:37:30] <marostegui>	 \o/
[16:09:58] <jynus>	 we'll see if it was the only issue
[16:10:12] <marostegui>	 did it finish?
[16:10:20] <jynus>	 it is catching up now
[16:10:24] <marostegui>	 coolio
[16:10:55] <jynus>	 that means it would have worked for 6 hours without breaking
[16:11:32] <marostegui>	 it was breaking every 24h so...
[16:11:35] <marostegui>	 we will see
[16:11:53] <jynus>	 let's do something, you work on the 7th, right?
[16:12:00] <marostegui>	 yep
[16:12:17] <jynus>	 check it then (let's down time it on icinga)
[16:12:25] <jynus>	 and note the exact row and table it breaks
[16:12:29] <jynus>	 if it does
[16:12:32] <marostegui>	 sounds good
[16:12:36] <jynus>	 but forget otherwise
[16:13:07] <jynus>	 I will ack it until monday
[16:13:11] <marostegui>	 cool
[16:13:14] <jynus>	 but remember to check it mid-week
[16:13:18] <marostegui>	 I will keep an eye on it the 7th
[16:13:19] <marostegui>	 yeah
[16:15:10] <marostegui>	 Going to log off - have a good vacations, see you on Monday :)
[16:15:17] <jynus>	 bye!, same
[16:15:21] <marostegui>	 o/
[17:23:59] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025#3813626 (10Lydia_Pintscher)
[18:19:40] <Amir1>	 marostegui: Around? https://gerrit.wikimedia.org/r/#/c/395584/
[18:20:08] <Amir1>	 This will increase size of the wbc_entity_usage tables but not much
[18:23:37] <marostegui>	 Amir1: Thanks for the heads up, how much is not too much? :)
[18:23:50] <Amir1>	 10% 
[18:24:00] <Amir1>	 No one uses descriptions in lua
[18:24:25] <marostegui>	 Ah right :)
[18:24:27] <marostegui>	 Thanks!
[19:46:57] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2044 - https://phabricator.wikimedia.org/T181779#3814091 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete
[19:49:28] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2044 - https://phabricator.wikimedia.org/T181779#3814099 (10Marostegui) Thanks - it is rebuilding correctly: ``` root@db2044:~# hpssacli controller all show config  Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380264FFFB0)      Port Name: 1I     Port...
[20:06:19] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-revi: Renaming 분당선M: supervision needed - https://phabricator.wikimedia.org/T182143#3814136 (10revi)
[20:07:32] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025#3814158 (10Addshore) @lydia if we want to do this we can easily :)
[20:22:54] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-revi: Renaming 분당선M: supervision needed - https://phabricator.wikimedia.org/T182143#3814207 (10Peachey88)