[04:36:25] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [04:37:26] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [05:07:30] 10DBA, 10Patch-For-Review: Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234300 (10Marostegui) This was done read only start: 05:00:17 read only stop: 05:00:43 Total read only time: 26 seconds [05:07:46] 10DBA, 10Patch-For-Review: Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234300 (10Marostegui) [05:09:42] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [05:22:49] 10DBA: Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234300 (10Marostegui) 05Open→03Resolved [05:22:51] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [05:22:53] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [05:24:05] 10DBA, 10Operations: decommission db1070.eqiad.wmnet - https://phabricator.wikimedia.org/T235464 (10Marostegui) [05:24:14] 10DBA, 10Operations: decommission db1070.eqiad.wmnet - https://phabricator.wikimedia.org/T235464 (10Marostegui) a:03Marostegui [05:24:48] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [05:24:59] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [05:44:52] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for banwiki - https://phabricator.wikimedia.org/T234770 (10Marostegui) a:05Marostegui→03None db1124, db2094, labsdb1009, labsdb1010, labsdb1011, labsdb1012 are clean. I have created the database on all... [06:32:58] 10DBA: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T235469 (10Marostegui) [06:33:18] 10DBA: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T235469 (10Marostegui) p:05Triage→03Normal [06:33:35] 10DBA: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T235469 (10Marostegui) [06:33:37] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [06:35:06] 10DBA, 10Operations: decommission db1070.eqiad.wmnet - https://phabricator.wikimedia.org/T235464 (10Marostegui) p:05Triage→03Normal [06:37:20] 10DBA, 10Operations, 10Patch-For-Review: decommission db1070.eqiad.wmnet - https://phabricator.wikimedia.org/T235464 (10Marostegui) [06:38:01] 10DBA, 10Operations, 10Patch-For-Review: decommission db1070.eqiad.wmnet - https://phabricator.wikimedia.org/T235464 (10Marostegui) [07:41:48] 10DBA, 10DC-Ops, 10decommission, 10Goal: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `dbstore2001.codfw.wmnet` - dbstore2001.codfw.wmnet (**PASS**) - Dow... [07:45:25] 10DBA, 10DC-Ops, 10decommission, 10Goal, 10Patch-For-Review: Decommission dbstore1001, dbstore2001, dbstore2002 - https://phabricator.wikimedia.org/T220002 (10Marostegui) [07:52:10] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui) [07:52:12] 10DBA, 10Operations: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234800 (10Marostegui) [08:30:19] backups seem to be expressing quantum properties: whenever I forget about them they work, but when I check them they fail [08:32:41] hahahaha [08:32:44] they miss you [10:13:34] https://phabricator.wikimedia.org/T234900#5574619 [11:10:11] the first size is...936GB? [11:10:18] Or 9.3TB? [11:10:25] I am getting lost with such a big number :) [11:29:39] ~1TB, as expected on our planning [11:29:48] :) [11:46:21] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [13:28:46] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for banwiki - https://phabricator.wikimedia.org/T234770 (10JHedden) a:03JHedden [13:53:09] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for banwiki - https://phabricator.wikimedia.org/T234770 (10JHedden) 05Open→03Resolved `name="Updates on labsdb10{09,10,11,12}" $ sudo /usr/local/sbin/maintain-replica-indexes --database banwiki --debug... [16:37:23] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): CloudVPS: m5-master databases for openstack may require re-encoding - https://phabricator.wikimedia.org/T234830 (10Andrew)