[03:56:32] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2703708 (10Bawolff) In case its helpful here: While working on a bug about what tables to classify for labs, I categorized the db tables at T103011#3536648... [04:58:11] 10DBA, 10Cloud-Services: Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3536665 (10Marostegui) Hi, Let the DBAs know once the database is actually created as it is not yet there. Not for DBAs, we need to ALTER the tables to get the PKs in place as the patch to... [05:05:21] 10DBA, 10Wikimedia-Site-requests: Global rename of Darwinius → DarwIn: supervision needed - https://phabricator.wikimedia.org/T173159#3536669 (10Marostegui) Hello @RuyP want to try it Tuesday at 9AM UTC? (or earlier if you like?) [05:13:55] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3536682 (10Marostegui) We might need to increase s7 buffer pool, as it is not getting a lot more delayed, but also not catching up: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=6&fullscreen&... [06:54:20] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536690 (10Marostegui) [07:43:28] 10DBA, 10Wikimedia-Site-requests, 10Tracking: Database table cleanup (tracking) - https://phabricator.wikimedia.org/T18660#3536706 (10Marostegui) [07:43:30] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3536703 (10Marostegui) 05Open>03Resolved cx_drafts table has been dropped on the master. [08:19:30] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3515549 (10Marostegui) a:03Marostegui [09:15:34] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536842 (10jcrespo) > Stop 's4' thread on db1095 and note down position and binlog That is wrong, and you are causing unnecessary lag on labs. Please start replication. [09:16:47] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536845 (10Marostegui) >>! In T172996#3536842, @jcrespo wrote: >> Stop 's4' thread on db1095 and note down position and binlog > > That is wrong, and you are causing unnecessary lag on labs. Please start re... [09:20:10] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536862 (10Marostegui) >>! In T172996#3536842, @jcrespo wrote: >> Stop 's4' thread on db1095 and note down position and binlog > > That is wrong, and you are causing unnecessary lag on labs. Please start re... [09:21:15] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536863 (10jcrespo) > any unexpected issues when loading on the other host Such as...? Will you do the same on mediawiki core? [09:22:40] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536864 (10Marostegui) >>! In T172996#3536863, @jcrespo wrote: >> any unexpected issues when loading on the other host > > Such as...? Such as mydumper not saving correctly the binlog position/ file on th... [09:22:46] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536865 (10jcrespo) Did you get a different number than the tool did automatically? ``` root@db1102:/srv/tmp/s4/s4$ grep -A3 s4 metadata Connection name: s4 Host: db1064.eqiad.wmnet... [09:23:52] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536867 (10Marostegui) >>! In T172996#3536865, @jcrespo wrote: > Did you get a different number than the tool did automatically? > ``` > root@db1102:/srv/tmp/s4/s4$ grep -A3 s4 metadata > Connection... [09:27:45] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536885 (10jcrespo) If you do not trust mydumper, we would want to know ASAP, thus what a better way than using it in a non-critical place and breaking or running a compare tool? One of our goals is to "test... [09:35:14] 10DBA, 10Patch-For-Review: Migrate s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996#3536887 (10Marostegui) >>! In T172996#3536885, @jcrespo wrote: > If you do not trust mydumper, we would want to know ASAP, thus what a better way than using it in a non-critical place and breaking or running... [10:41:36] 10DBA, 10Wikimedia-Site-requests: Global rename of Darwinius → DarwIn: supervision needed - https://phabricator.wikimedia.org/T173159#3537038 (10RuyP) >>! In T173159#3536669, @Marostegui wrote: > Hello @RuyP want to try it Tuesday at 9AM UTC? (or earlier if you like?) @Marosterogui. Ok. :) See you tomorrow at... [12:38:46] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3537309 (10Marostegui) [12:39:22] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3533347 (10Marostegui) a:03Cmjohnson This host is now ready for the remaining steps from @Cmjohnson [14:06:41] https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&from=now-90d&to=now&panelId=32&fullscreen&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13316 [14:06:44] vs [14:07:06] https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&from=1495548417863&to=1503324417863&panelId=32&fullscreen&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2002&var-port=13316 [14:19:05] wow [14:24:57] nice [14:25:07] what is that difference? [14:29:13] 90 days ago it was running multisource and now it is running multi instance [14:31:10] not nice [14:31:19] one host has faster disks than the other [14:32:13] ah, I assumed it was the same host! [14:32:19] Didn't check it [14:32:49] dbstore2002 has less hits though, so it could "make" sense [14:32:59] no, even before [14:33:11] the latency is consistent [14:33:53] someties better, sometimes worse [14:34:15] but the cloud is much disperse and in average higher on 2001 [15:09:29] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3538008 (10Marostegui) I am checking db1041 against db1079. Although looks like this shard is pretty consisten across slaves