[06:38:10] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984767 (10Marostegui) [06:41:32] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984768 (10Marostegui) My proposal is: Replace db2067 and db2060 (160GB) in s6 with a large host. This is the current status of s6 ``` 's6' => [ '... [06:43:34] 10DBA, 10Data-Services: Re-institute query killer for the analytics WikiReplica - https://phabricator.wikimedia.org/T183983#3984769 (10Marostegui) Reporting status: So far only queries on Query state and running longer than 14400 seconds have been killed over night (two queries) so, so far so good. [09:12:27] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984885 (10jcrespo) I would like to focus on s4. s4 needs better hardware than s6, then do: ``` 'db2051' => 0, # B8 2.9TB 160GB, master - 'db2037' =>... [09:13:28] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984891 (10Marostegui) >>! In T187722#3984885, @jcrespo wrote: > I would like to focus on s4. s4 needs better hardware than s6, then do: I was doubting betwee... [09:17:51] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984897 (10jcrespo) We also have backups from the host that crashed itself on dbstore2001, I think, we could use them to reconstruct m5 without touching the ma... [09:19:46] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984899 (10jcrespo) >>! In T187722#3984891, @Marostegui wrote: >>>! In T187722#3984885, @jcrespo wrote: >> I would like to focus on s4. s4 needs better hardwar... [09:22:53] 10DBA, 10Operations, 10ops-codfw: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3984900 (10Marostegui) >>! In T187722#3984897, @jcrespo wrote: > We also have backups from the host that crashed itself on dbstore2001, I think, we could use t... [09:46:01] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#3984921 (10Marostegui) [10:04:06] 10DBA: Decommission db2030 - https://phabricator.wikimedia.org/T187768#3984964 (10Marostegui) p:05Triage>03Normal [10:05:17] 10DBA: Decommission db2030 - https://phabricator.wikimedia.org/T187768#3984964 (10Marostegui) [10:05:19] 10DBA, 10Goal, 10Patch-For-Review: Decommission database hosts <= db2031 (tracking) - https://phabricator.wikimedia.org/T176243#3984980 (10Marostegui) [10:20:43] 10DBA, 10Patch-For-Review: Decommission db2030 - https://phabricator.wikimedia.org/T187768#3985000 (10Marostegui) [10:45:24] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3985041 (10Marostegui) [10:45:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3985042 (10Marostegui) [11:01:00] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3985083 (10jcrespo) Aside from m5, what is the other host for? x1 or m2? [11:01:56] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3985086 (10Marostegui) So I think db2037 -> m5 db2044 -> x1/m2 no? [11:02:15] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3985088 (10Marostegui) m2, sorry [11:04:24] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#3985090 (10Marostegui) s2 progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore1002 [... [11:04:33] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3985091 (10Marostegui) s2 progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102... [11:04:43] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3985092 (10Marostegui) s2 progress: [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1102 [] dbstore1002 [x] dbstore1001 (b... [11:05:04] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#3985093 (10Marostegui) [11:05:18] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3985094 (10Marostegui) [11:06:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3985098 (10Marostegui) [11:09:34] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3985108 (10Marostegui) [11:09:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3985110 (10Marostegui) [11:29:15] 10DBA, 10Patch-For-Review: Decommission db2030 - https://phabricator.wikimedia.org/T187768#3985149 (10Marostegui) [13:01:38] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for romdwikimedia - https://phabricator.wikimedia.org/T187774#3985290 (10Urbanecm) p:05Triage>03Low [13:08:56] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for romdwikimedia - https://phabricator.wikimedia.org/T187774#3985325 (10Marostegui) Let us know when the wiki is created to filter it on labs and apply (or check if we need to apply): T187089 T185128 T153182 [13:56:05] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3985543 (10Marostegui) Revision table is done. Next: watchlist (which is in a pretty good state, so it shouldn't take long) [16:41:39] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986351 (10Cmjohnson) [16:43:08] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3925325 (10Cmjohnson) @Marostegui This server only has 2 4TB disks, with no raid card. This will need a software raid. Let me know if you want to reconsid... [16:45:10] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986376 (10jcrespo) ^that was my main reason to object to not call it db*, all db* hosts have a hardware raid and are, to some extent, interchangable. [17:00:11] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986472 (10Cmjohnson) It's not too late to rename it to tendril1001...easy dns change right now [17:05:45] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986520 (10Marostegui) >>! In T185788#3986376, @jcrespo wrote: > ^that was my main reason to object to not call it db*, all db* hosts have a hardware raid a... [17:05:59] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986522 (10jcrespo) no, tendril is definitely not ok. [17:07:06] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986529 (10jcrespo) I did not reopen the discussion this, Chris did. [17:21:02] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3986625 (10Marostegui) [17:21:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3986626 (10Marostegui) [17:27:43] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986656 (10Marostegui) @Cmjohnson, please proceed with db1115 as @jcrespo and myself agreed on that hostname yesterday in our weekly meeting. [17:55:46] 10DBA, 10PAWS: Analyse PAWS query killer - https://phabricator.wikimedia.org/T187818#3986819 (10Chicocvenancio) [18:09:44] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986867 (10Cmjohnson) @marostegui Okay, since I cannot do standard DB raid...any suggestions? [18:10:53] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986868 (10jcrespo) Do nothing, we (the recipe) will install the RAID1 in software. [18:12:15] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986878 (10jcrespo) ``` $ git grep db1115 modules/install_server/files/autoinstall/netboot.cfg: db1115|db2093) echo partman/raid1-gpt.cfg ;; \ ``` [18:14:30] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986883 (10Cmjohnson) [18:21:19] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3986893 (10jcrespo) [19:04:28] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3987008 (10jcrespo) [19:04:32] 10DBA, 10Goal, 10Patch-For-Review: Decommission database hosts <= db2031 (tracking) - https://phabricator.wikimedia.org/T176243#3987007 (10jcrespo) [19:05:24] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3854215 (10jcrespo) [19:05:29] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3987011 (10jcrespo) [20:01:43] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3987221 (10jcrespo) [20:01:45] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) - https://phabricator.wikimedia.org/T187722#3987220 (10jcrespo) [20:02:00] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3854215 (10jcrespo)