[04:25:50] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables after the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Niharika) @Huji Is there a similar task for tracking table bloat from login attempts data? [05:07:57] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6291179, @WDoran... [05:16:01] 10DBA, 10Patch-For-Review: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Marostegui) @jcrespo @Kormat let's do this Wednesday 15th July at 06:00 AM UTC? [05:17:14] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) As agreed on IRC, I am going to take db1084 to use it to upgrade m2, then old m2 master will be used to upgrade m3, old m3 master will be used to upgrade m5, and finally, that old m5 master will be use... [05:21:01] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) [05:21:23] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) p:05Triage→03Medium [05:22:01] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) [05:34:55] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) 05Stalled→03Open [05:57:11] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Dzahn) >>! In T255715#6284613, @Marostegui wrote: > I would suggest we start by renaming the tables... [05:58:56] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) There seem to be two parallel conversations, one here and one on the patch :-) I don't m... [06:13:44] 10DBA, 10Patch-For-Review: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10jcrespo) Ok to me. [06:37:38] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1084.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202007090637_m... [06:58:20] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1084.eqiad.wmnet'] ` and were **ALL** successful. [07:04:00] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1084.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202007090703_m... [07:24:15] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1084.eqiad.wmnet'] ` and were **ALL** successful. [07:26:47] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [08:09:32] 10DBA: Choosing a wrong host with transfer.py produces an "ERROR: The specified source path X doesn't exist on Y" - https://phabricator.wikimedia.org/T256951 (10jcrespo) 05Open→03Resolved a:03jcrespo I believe this to be fixed. Updated package will come soon for production. [08:09:35] 10DBA, 10Google-Summer-of-Code (2020), 10Patch-For-Review: GSoC 2020 Proposal: Improve the framework to transfer files over the LAN - https://phabricator.wikimedia.org/T248256 (10jcrespo) [08:09:47] 10DBA: Choosing a wrong host with transfer.py produces an "ERROR: The specified source path X doesn't exist on Y" - https://phabricator.wikimedia.org/T256951 (10jcrespo) a:05jcrespo→03Privacybatm [08:15:44] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) I don't think this setup is adequate for testing paralelism, given we only have 1 host to transfer to (in parallel). I believe this could be way more interesting... [08:39:55] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10Marostegui) a:03Marostegui [08:55:39] 10DBA, 10Goal: Expand database provisioning/backup service to accomodate for growing data, redundancy and performance needs - https://phabricator.wikimedia.org/T257551 (10jcrespo) [08:55:47] 10DBA, 10Goal: Expand database provisioning/backup service to accomodate for growing data, redundancy and performance needs - https://phabricator.wikimedia.org/T257551 (10jcrespo) p:05Triage→03High [08:58:51] 10DBA, 10Goal: Expand database provisioning/backup service to accomodate for growing data, redundancy and performance needs - https://phabricator.wikimedia.org/T257551 (10jcrespo) [09:18:06] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1017.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202007090917_maro... [09:32:23] 10DBA, 10Operations, 10CAS-SSO, 10Patch-For-Review, 10User-jbond: Request new database for idp-test.wikimedia.org - https://phabricator.wikimedia.org/T256120 (10Marostegui) @jbond were you able to check if the connection works fine and the application can reach the DB via the proxy? [09:34:16] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [09:38:40] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) s5 eqiad progress [] labsdb101... [09:38:54] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) [09:39:01] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1017.eqiad.wmnet'] ` and were **ALL** successful. [09:43:04] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10Marostegui) [09:44:11] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) >>! In T254979#6292417, @jcrespo wrote: > I don't think this setup is adequate for testing paralelism, given we only have 1 host to transfer to (in parallel).... [10:00:59] jynus: feeling like stopping your clinic duty to come to a meeting? :p [10:13:34] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1131 - https://phabricator.wikimedia.org/T257253 (10Cmjohnson) {F31921902}. @Jclark-ctr TSR report is attached and will email [11:08:52] 10DBA, 10Patch-For-Review: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) > But for the coming week, I would like to focus on the automatic folder creation (/var/lib/transferpy) and changing the temp files to them Sounds good to me, cr... [11:14:48] 10DBA, 10Goal: Expand database provisioning/backup service to accomodate for growing capacity and high availability needs - https://phabricator.wikimedia.org/T257551 (10jcrespo) [11:33:27] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10Marostegui) [11:35:23] jynus: can you confirm how many hosts you'd need to order in q1? [11:35:26] regular db specs [11:35:37] regular db only 1 per dc [11:35:42] gotcha thanks [11:36:12] line 50 and 51 [11:36:19] haha I was just writing that :) [11:36:36] you know I don't mind requesting them on my own, ok? [11:36:43] after hearing mgr [11:36:44] sure, I am emailing willy first [11:36:49] sure, as we agreed [11:36:51] to see what is easier for you [11:36:54] and him [11:37:06] you know it is a lot of overhead for just 1 host [11:37:29] and if we do 2 batches at the same time we can handle them 50/50 [11:59:42] ok to reboot tendril/dbmonitor1001 for a kernel update? [12:01:05] yep from my side [12:02:14] ack, unless there are any further objections I'll start in 10m, then [12:03:42] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10Marostegui) s1 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003... [12:10:03] 10DBA, 10Core Platform Team, 10Schema-change, 10User-DannyS712: slot_revision_origin_role should be a UNIQUE INDEX - https://phabricator.wikimedia.org/T256841 (10Marostegui) I would like to hear #core_platform_team opinion on this. As this table is quite recent and I don't know what was the reason to defin... [12:12:14] 10DBA, 10Schema-change, 10User-DannyS712: iwlinks indexes should be UNIQUE INDEXes - https://phabricator.wikimedia.org/T256842 (10Marostegui) What'd be the benefit of doing this change across the fleet? The table is very small but still big enough that this would require host-by-host changes, so what's the b... [12:12:16] ^I am confused by that request, 2 columns being unique doesn't mean the 3 are [12:12:26] the one on slot_revision [12:12:49] but I don't know what those tables do, so I would done the same, ask CPT [12:12:53] yeah [12:13:01] *have done [12:13:09] It confused me as well, and that table is really new, so not sure what's the motivation [12:13:52] maybe the logic is right and there are issues with duplication [12:14:08] but if that's the case, that needs more background [12:14:10] let's see what CPT says, they will know the contect better [12:14:11] yeah [12:14:14] *context [12:15:43] 10DBA, 10Schema-change, 10User-DannyS712: iwlinks indexes should be UNIQUE INDEXes - https://phabricator.wikimedia.org/T256842 (10DannyS712) >>! In T256842#6293177, @Marostegui wrote: > What'd be the benefit of doing this change across the fleet? The table is very small but still big enough that this would r... [12:25:45] 10DBA, 10Schema-change, 10User-DannyS712: iwlinks indexes should be UNIQUE INDEXes - https://phabricator.wikimedia.org/T256842 (10Marostegui) There are benefits and drawbacks on having UNIQUE indexes, and I would have expected some more background on what type of data the table stores and how unique that dat... [12:34:48] 10DBA, 10Core Platform Team, 10Schema-change, 10User-DannyS712: iwlinks indexes should be UNIQUE INDEXes - https://phabricator.wikimedia.org/T256842 (10Marostegui) Let's get some input from #core_platform_team [12:53:40] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables after the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Huji) @Niharika not yet. We are still two steps away from that data getting into CU tables ([[https://gerrit.wikimedia.org... [16:00:05] 10DBA, 10Core Platform Team, 10Schema-change, 10User-DannyS712: slot_revision_origin_role should be a UNIQUE INDEX - https://phabricator.wikimedia.org/T256841 (10daniel) Since it is already guaranteed to be unique, it's probably unnecessary to explicitly define it as unique. I once thought that doing so wo... [17:51:57] 10DBA, 10Google-Summer-of-Code (2020): Create temp and config directories at the installation time of transferpy deb package - https://phabricator.wikimedia.org/T257599 (10Privacybatm) [17:55:09] 10DBA: Create temp and config directories at the installation time of transferpy deb package - https://phabricator.wikimedia.org/T257599 (10Privacybatm) [17:58:02] 10DBA: Create more tests for transfepy package - https://phabricator.wikimedia.org/T257600 (10Privacybatm) [18:02:53] 10DBA: transferpy 1.0 release - https://phabricator.wikimedia.org/T257601 (10Privacybatm) [18:07:23] 10DBA: Make transferpy configurable using a configuration file - https://phabricator.wikimedia.org/T257602 (10Privacybatm) [18:17:17] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables after the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Niharika) >>! In T257223#6293268, @Huji wrote: > @Niharika not yet. We are still two steps away from that data getting int... [18:47:01] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1131 - https://phabricator.wikimedia.org/T257253 (10Jclark-ctr) @Marostegui Tsr report showed a few more errors and dell would like to address. what day works best to schedule downtime? ` Good morning John, Per our phone conversations this morning, w... [19:26:07] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1131 - https://phabricator.wikimedia.org/T257253 (10Marostegui) Thanks @jclark-ctr. We need to schedule a maintenance window as this is an active master. I will get that done next week and let you know when you can power off the host and replace the board....