[03:02:32] 10DBA, 10Operations: File space alert for db1028 - https://phabricator.wikimedia.org/T169294#3393763 (10Andrew) [04:47:35] 10DBA, 10Operations: File space alert for db1028 - https://phabricator.wikimedia.org/T169294#3393798 (10Marostegui) p:05Triage>03Normal a:03Marostegui Hi Andrew, Thanks for the ticket. You are indeed correct, this is MySQL usage. The reason for this sudden growth of disk space is the ALTER table going... [04:54:41] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3393804 (10Marostegui) [06:13:23] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3393867 (10Marostegui) [06:20:55] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3393868 (10Marostegui) From s2, only the primary master is pending. I have done a test with some slaves with the same hardware without depooling them, and th... [07:41:56] 10DBA, 10Operations: File space alert for db1028 - https://phabricator.wikimedia.org/T169294#3393942 (10Marostegui) And the alter for the big table finished and space recovered: ``` root@db1028:~# df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 1.7T 1.4T... [08:15:22] 10DBA, 10Operations: File space alert for db1028 - https://phabricator.wikimedia.org/T169294#3394003 (10Marostegui) 05Open>03Resolved [08:45:28] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3394046 (10akosiaris) [08:53:49] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10jcrespo) We should not enable active-active on dbtree (or enable it failing, as it is the current case). Dbtree database backend is db1011, which is only on eqi... [08:56:36] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3394080 (10jcrespo) [09:03:32] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3394123 (10Marostegui) [09:05:35] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3394128 (10Marostegui) s6: It is all done, I believe we can also alter the master early in the morning, as I have done a test with a less powerful host witho... [10:50:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3394346 (10Marostegui) s5: All done but the primary master. Probably this one can also be altered on the master on an early morning. I have done tests with a... [10:50:20] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3394347 (10Marostegui) [11:00:15] 10DBA, 10Operations, 10Patch-For-Review: Prepare mysql hosts for stretch - https://phabricator.wikimedia.org/T168356#3394408 (10jcrespo) 05Open>03Resolved This is in no way a closed issues, but the initial scope is covered- pending tidying up puppet and hiera code. But the support is working, at least as... [11:54:22] 10DBA, 10Epic: Meta ticket: The future of multi source replication slaves vs multi instance ones. - https://phabricator.wikimedia.org/T159423#3394606 (10Marostegui) Does it make sense to keep this open if we have already decided to go for multi-instance? [12:23:37] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 10RfC: Should we bump MediaWiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3394750 (10daniel) During the ArchCom meeting on June 28, it was agreed for this RFC to enter the Last Call period. If no pertinent issues remain... [12:38:27] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 10RfC: Should we bump MediaWiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3394796 (10jcrespo) Not technically part of the the RFC, but the closer we get to 5.7, the closer this will become a problem: T108255 This is no... [12:43:18] I think grafana no longer works on the new labsdbs, probably due to new puppet code [12:43:31] which check it when I finish what I am doing [13:17:31] labsdb1009 is fixed now, the other will be fixed when puppet reruns [13:31:38] so it was the socket thingy only? [13:32:33] yes [13:38:21] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3395043 (10Marostegui) [13:58:08] 10DBA, 10Wikidata, 10Performance: slow master queries on Wikibase\Client\Usage\Sql\EntityUsageTable::getAffectedRowIds - https://phabricator.wikimedia.org/T169336#3395074 (10jcrespo) [14:28:09] 10DBA, 10Operations: File space alert for db1028 - https://phabricator.wikimedia.org/T169294#3395189 (10Andrew) thanks! [15:06:12] 10DBA, 10Operations, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3395348 (10madhuvishy) @jcrespo Apologies for the delay. Can we start with just labsdb1005 first, and attempt to do it Wednesday July 5, and labsdb1004 on Thursday July 6, provided the... [15:12:32] 10DBA, 10Operations, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3395356 (10Cmjohnson) @madhuvishy I am out all next week and will be back July 11. [15:14:39] 10DBA, 10Operations, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3395358 (10madhuvishy) @Cmjohnson Okay thanks for letting me know, I'll schedule the labsdb1001 and 1003 reboots (the ciscos), for after you are back then. When are you in the DC (from... [15:16:21] 10DBA, 10Operations, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3395385 (10Cmjohnson) @madhuvishy I typically get the DC around 1400UTC (10am EST). [17:44:57] marostegui, jynus: FYI T169355 (adding it here just because wikibugs quit right before it) [17:44:57] T169355: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355 [17:53:14] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395711 (10Marostegui) @Cmjohnson you still in the DC? [17:53:47] volans: I have asked chris if he is still in the dc, maybe he can replace the disk now [17:55:54] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395717 (10Cmjohnson) @marostegui I am not [18:03:18] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395781 (10Marostegui) @Cmjohnson if it helps, there are some hosts that are ready to be decommissioned which have 600GB disks which are probably old though: T166486 T164702 [18:04:08] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395785 (10jcrespo) p:05Triage>03High [19:20:28] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395891 (10Cmjohnson) @marostegui the disk has been swapped with the last new spare disk on-site. [19:21:18] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3395892 (10Cmjohnson) Currently rebuilding Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 0 WWN: 5000C500437173D8 Sequence Number: 11... [20:19:55] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3396020 (10Marostegui) >>! In T169355#3395891, @Cmjohnson wrote: > @marostegui the disk has been swapped with the last new spare disk on-site. Thanks Chris! Should we order more spares or how is this usuall... [21:47:56] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3396274 (10Volans) Rebuild completed, RAID back to optimal. There are 2 disks with predictive failure that might fail sooner or later ``` $ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli === Rai... [21:58:37] 10DBA: Truncate table "l10n_cache" on all wmf sites - https://phabricator.wikimedia.org/T169375#3396283 (10demon) [22:12:01] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1052 - https://phabricator.wikimedia.org/T169355#3396375 (10Marostegui) 05Open>03Resolved a:03Cmjohnson Great!! Thanks! I will close this for now, and we will check if we need to buy more disks next week! Thanks a lot Chris!