[02:11:11] 10DBA, 10Operations, 10Release-Engineering-Team, 10cloud-services-team, 10Chinese-Sites: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3899755 (10Shizhao) [04:12:41] 10DBA, 10Operations, 10Release-Engineering-Team, 10cloud-services-team: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3899801 (10Liuxinyu970226) @Shizhao I'm sorry but Jcrespo hasn't decision about zhwiki here [06:15:49] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3899829 (10Marostegui) [06:16:17] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3899841 (10Marostegui) p:05Triage>03Normal [06:16:45] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2036 - https://phabricator.wikimedia.org/T184836#3899843 (10Marostegui) p:05Triage>03Normal [06:29:33] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3899844 (10Papaul) @Marostegui sorry but we don't have any used BBU from a decommissioned host that we can use . (we have no decommissioned HP servers) [06:35:34] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3899858 (10Marostegui) Thanks @Papaul - I have checked the hosts that will soon be decommissioned and none of them are HP. @RobH any ideas on what can we do about this? [07:18:24] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3899874 (10Marostegui) [07:35:00] 10DBA, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3899879 (10Marostegui) I have set db1087 as vslow in s8 instead of db1063, which is the first step to move db1063 as vslow in s6, so we can get rid of db1030. I will reimage db1063 as stretch and use 10.1 for it //cc @... [08:51:15] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3899909 (10Marostegui) s5 master is done [08:51:41] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3899910 (10Marostegui) [08:55:21] 10DBA, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3899916 (10jcrespo) Is is similar to https://gerrit.wikimedia.org/r/#/c/399792/3/wmf-config/db-eqiad.php or de we need an update? [08:58:34] 10DBA, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3899922 (10Marostegui) It is exactly that :) [09:47:02] I've just uploaded 10.1.30 for jessie [09:47:09] \o/ [09:47:44] I have proxysql_2.0~lab too [09:48:15] but I do not think I am going to upload it, I will side-load it because it is experimental [09:48:31] hehe yeah, I read you were reading proxysql code during the weekend [09:48:57] in theory that supports TLS1.2 [09:49:10] but I am not sure it works properly yet [11:55:13] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3900321 (10Marostegui) labsdb1003 RAID policy started to fail and it is now on WT instead of WB. P... [12:00:47] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3900336 (10Marostegui) Archive table is now fixed across all the servers. Next: change_tag [13:29:51] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3900477 (10Marostegui) Maybe we should force this host to be WB even without the BBU to make sure it catches up: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2F... [16:38:02] 10DBA, 10Operations, 10ops-codfw: Failed BBU on db2033 (x1 master) - https://phabricator.wikimedia.org/T184888#3900967 (10Marostegui) The server kept lagging. I have forced the controller to go to WriteBack temporarily till we decide how to proceed with this host. ``` root@db2033:~# hpssacli controller all s... [16:50:27] 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team: Configure Toolforge replica views and dumps for the new MCR tables - https://phabricator.wikimedia.org/T184446#3883142 (10ArielGlenn) slot_roles and content_models can be dumped as is by adding them to the list of tables to be dumped: h... [16:52:40] !log upgrade and restart db2018 [16:52:40] jynus: Not expecting to hear !log here [17:18:33] 10DBA: Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029 - https://phabricator.wikimedia.org/T184090#3901088 (10jcrespo) [17:18:54] 10DBA: Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029 - https://phabricator.wikimedia.org/T184090#3872109 (10jcrespo) a:03jcrespo [17:21:11] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2036 - https://phabricator.wikimedia.org/T184836#3898087 (10jcrespo) As a heads up, this is now the s3 master. [19:40:15] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2036 - https://phabricator.wikimedia.org/T184836#3901404 (10Marostegui) [22:00:45] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3901520 (10bd808) >>! In T142807#3900321, @Marostegui wrote: > @bd808 any objection to stop MySQL...