[08:35:04] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: s5 primary master db1063 crashed - https://phabricator.wikimedia.org/T180714#3771983 (10Marostegui) The schema change revert has finished on codfw master and replication has been started correctly. The slaves are now getting the alters, and the...
[08:35:25] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: s5 primary master db1063 crashed - https://phabricator.wikimedia.org/T180714#3771984 (10Marostegui)
[10:02:56] <wikibugs>	 10DBA, 10Cloud-Services, 10Cloud-VPS, 10Operations, 10Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3771994 (10Marostegui) >>! In T157359#3146925, @jcrespo wrote: > Ok, now puppet works, but either it puppet needs more work or it fails silently-thi...
[12:20:48] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3772114 (10jcrespo) db2085 is going quite short in space available because of the multiple instances + wikidata alters. Is it fully compressed? https://grafana.wikimedia.org/dashboard/file/server-board.j...
[12:44:26] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3772116 (10Marostegui) Probably not by looking at the size of s5 specially, might have left forgotten: ``` 1.7T sqldata.s3 1.7T sqldata.s5 ```
[12:54:41] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3772120 (10Marostegui) Comparing it to db2086.s5, looks like it needs some compressing: ``` root@db2085:/srv/sqldata.s5/wikidatawiki# ls -lh wb_terms.ibd -rw-rw---- 1 mysql mysql 539G Nov 17 09:19 wb_ter...
[13:03:20] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3772133 (10Marostegui) db1101.s7 finished its compression.
[13:07:45] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3772146 (10Marostegui) s5 will be impossible to be done for as long as we keep the master with ROW. As we cannot alter slaves as the...
[13:08:21] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3772147 (10Marostegui)
[15:52:07] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3772259 (10Marostegui) The alters finished in db2085 (and it is catching up) so I am going to compress s3 and s5 there.