[07:00:53] 10DBA, 10Data-Services: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4196167 (10jcrespo) [08:31:49] 10DBA, 10Data-Services, 10Patch-For-Review: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4196270 (10Marostegui) For what is worth. There was a heavy alter table running on Commons.logging for the last 24-36h which is a pretty big table so that might have contributed to maximiz... [08:44:52] 10DBA, 10Data-Services, 10Patch-For-Review: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4196167 (10Ladsgroup) Also, I have been deleting rows from logging tables in large wikis and changing values in wb_terms of wikidatawiki. This might contribute to the load but they will go... [09:36:29] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4196559 (10jcrespo) Migrated db1072's replicas to db1077, with the following coordinates: ``` db1072: db1072-bin.000860:589708946 / db1075-bin.002777:647785363 db1077: db1077-bin.002831:968832211 ``` [14:41:45] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4197363 (10Papaul) a:05Papaul>03Marostegui @Marostegui replaced the disk with another disk. [14:43:01] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4197368 (10jcrespo) a:05Marostegui>03jcrespo Manuel is not around today, I am taking the task. [15:39:58] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4197580 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1072.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201... [16:00:13] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4197664 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1072.eqiad.wmnet'] ``` and were **ALL** successful. [16:17:40] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 2 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4197718 (10CCicalese_WMF) [16:20:54] 10DBA, 10MediaWiki-Platform-Team, 10Multi-Content-Revisions, 10Structured-Data-Commons, and 2 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4197738 (10CCicalese_WMF) [16:54:05] 10Blocked-on-schema-change, 10DBA, 10Structured-Data-Commons, 10Wikidata, 10Multi-Content-Revisions (MCR-SDC phase 1): DROP unused 'slots' table (WAS: In the slots table, replace slot_inherited with slot_origin) - https://phabricator.wikimedia.org/T190153#4197921 (10CCicalese_WMF) [16:55:42] 10DBA, 10Structured-Data-Commons, 10Wikidata, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), and 2 others: MCR schema migration stage 0: create tables - https://phabricator.wikimedia.org/T183486#4197937 (10CCicalese_WMF) [17:07:37] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4198044 (10jcrespo) @Papaul can you check the disk used? It says it has 300GB, either it is wrongly detected or a mistake (this host needs 600GB disks), and it failed becaue of that: ``` physicaldrive 1I:1:... [17:07:42] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T194103#4198048 (10jcrespo) a:05jcrespo>03Papaul [17:13:56] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4198086 (10jcrespo) db1072 is loading m3 from the last logical backup. [17:14:50] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1055 - https://phabricator.wikimedia.org/T194118#4198106 (10jcrespo) a:05jcrespo>03RobH @robh this can now proceed as usual, db1055 is a spare with notifications disabled. Please note that in case of reusing its parts, megaraid,1,megaraid... [17:15:58] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1055 - https://phabricator.wikimedia.org/T194118#4198112 (10jcrespo) [17:16:17] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1055 - https://phabricator.wikimedia.org/T194118#4189202 (10jcrespo) p:05Normal>03Low [17:47:38] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4198180 (10jcrespo)