[00:32:00] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) [00:37:24] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) [00:38:40] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) [01:29:32] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) [04:26:43] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10jcrespo) Hi, may I ask to mockup (although as realistically as possible), 1 or 2 rows per table to better understand the amount of space that will be neede... [05:11:27] 10DBA, 10Operations: Switchover s8 (wikidata) primary database master db1104 -> db1109 - 10th Sept @05:00 UTC - https://phabricator.wikimedia.org/T230762 (10Marostegui) This was done. read-only start: Tue Sep 10 05:00:47 UTC 2019 read-only stop: Tue Sep 10 05:02:14 UTC 2019 Total read-only time: 1 minute 27 s... [05:19:53] 10DBA, 10Operations: Switchover s8 (wikidata) primary database master db1104 -> db1109 - 10th Sept @05:00 UTC - https://phabricator.wikimedia.org/T230762 (10Marostegui) 05Open→03Resolved [05:30:31] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2047.codfw.wmnet - https://phabricator.wikimedia.org/T231852 (10Marostegui) [05:30:48] checking snapshot of s5 in codfw [05:31:36] thanks [05:31:45] and also thank you for waking up early for the failover :* [05:37:30] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [05:46:21] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [05:58:28] <3 "LOOKING FOR THE LOAD LISTS?" [06:02:54] load lists? [06:05:19] the new comment on db-eqiad.php [06:06:06] aaah [06:06:09] haha [08:51:28] marostegui: jynus did everything went fine with the switchover? Can I turn on write both for all of wikidata? [08:52:13] Amir1: yep! [08:52:19] YES [08:52:19] You want me to merge that puppet change? [08:53:00] marostegui: merge for now should be fine but we should not start it until I turn on write_both on all of wikidata [08:53:06] Amir1: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534790/ was it, right? [08:53:19] (It's absent right now) [08:54:47] marostegui: Can you also make the file as well? https://gerrit.wikimedia.org/r/c/operations/puppet/+/534790/#message-784603f89a26bf8af95d5d811df6aabc3ae635e3 [08:55:24] yep [08:55:36] just a touch and add that sentence? [08:55:56] it would be great if it has the same permissions as the other files in the directory so the maintenance script can write to it [08:56:05] yup [08:56:06] sure [08:57:10] Thank you! [08:57:57] Amir1: like this? [08:57:58] root@mwmaint1002:/var/log/wikidata# cat wikidata-rebuildItemTerms.log [08:57:58] Rebuilding Q501 till Q2000000 [08:58:42] yup [08:58:49] the last number is important [08:59:50] https://phabricator.wikimedia.org/T225056#5478094 [09:06:57] we may need to revisit compression on some wikidata tables [09:08:51] can you create a task so we don't forget? [09:11:58] marostegui: https://gerrit.wikimedia.org/r/#/c/535526/ this needs to be merged later today [09:12:18] It cannot be done now? [09:12:32] jynus: oh, once wb_terms is killed? or during migration? [09:12:53] well, preferibly weeks ago [09:13:01] marostegui: no, because if it starts and the config is set to write old, eveything in between will be lost [09:13:17] new master? [09:13:26] Amir1: gotcha, I have an interview later, then I will take a very long lunch break and be back around 5pm, as we have another master switchover at 6pm [09:13:35] (and I started at 6:15am today) [09:14:07] oh jeez [09:14:11] get rest [09:14:19] this can also go live tomorrow [09:14:33] Amir1: ah cool, let's merge it tomorrow morning then? [09:14:48] surio [09:15:05] great, ping me tomorrow when you want it merged ) [09:15:07] :) [09:15:22] will do [09:18:07] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) [09:19:05] <3 jaime [09:19:30] I may start working on that right now, for non production hosts (backups) [09:19:37] sweet! [09:19:56] please write down there the hosts that get done, for tracking [09:20:33] yeah, of course, if/when i do [09:20:39] thanks :* [09:20:45] we also have backed up 30 million files in the last 6 months [09:20:55] :O [09:34:45] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) a:03jcrespo I will do a test run on db2100:3318, to see how much time it may take currently. [09:34:54] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) p:05Triage→03Normal [10:18:37] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534386/ [10:39:51] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) [10:43:12] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) It took one hour, but the compression was worth it: ` -rw-rw---- 1 mysql mysql 8.5G Sep 10 09:38 wbt_item_terms.ibd -rw-rw---- 1 mysql mysql 36M Sep 10 08:13 wbt_property_terms.ibd -rw-rw---- 1 mysql mysql 4.2G Sep 10... [10:43:46] maybe I should do a marathon and try to do as much of that as possible before tomorrow [10:44:06] jynus: you can probably do codfw directly on the master [11:09:08] It's set to write both, the rows written got an increase: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=s8&var-role=All&from=now-1h&to=now [11:15:17] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) Compressing db1116:s8, however that may take a while as it has other uncompressed tables that are also being converted. [12:41:06] b6 pdu swap is over [13:23:48] cmjohnson1: not need to announce it on every chanlle, but it is appreciated that you did! [13:24:35] jynus after the last time where ppl were surprised about the move I wanted to make sure that it wasn't missed. [13:24:40] :-) [13:24:45] thanks [14:01:42] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) @jcrespo Sure. Here are a few samples from my local machine: - machine_vision_provider ` (13:48) vagrant@localhost:[commonswiki]> select * from... [14:16:55] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) Probably the most unusual thing going on in this extension, which I should mention specifically here, is the getTitlesWithUnreviewedLabels metho... [14:20:23] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10jcrespo) Please see the second part of my request for information on the comment above. [14:21:52] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Mholloway) @jcrespo Ah, sorry about that. I don't believe anything here needs to be private. [16:20:52] 10DBA, 10Operations, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui) This was done. Read-only starts: Tue Sep 10 16:10:39 UTC 2019 Read-only stops: Tue Sep 10 16:10:45 UTC 2019 Total read-only time:... [16:22:10] 10DBA, 10Operations, 10User-notice: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui) 05Open→03Resolved Thanks everyone who helped out, closing this! [16:22:13] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui)