[09:17:18] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10jcrespo) @Papaul sorry he will be away, I don't have a clear solution, but I believe any other place on the same row, but not the same rack as the others will do. Please note th... [09:18:35] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10jcrespo) [09:21:53] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10jcrespo) [09:31:20] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10jcrespo) Task has been created, @Marostegui will take care of creating the racking strategy on January, but if for some reason you are in a hurry (we are not), it will be distrib... [10:05:34] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10Dzahn) Added steps for a fail-over to a new server on https://wikitech.wikimedia.org/wiki/Phabricator#Simple_fai... [10:10:09] 10DBA: Move more wikis from s3 to s5 - https://phabricator.wikimedia.org/T226950 (10jcrespo) The largest tables on loginwiki are: ` user logging actor watchlist spoofuser ` Together they take 2.9 GB compressed, or the 96% of the total size compressed. Note that is small for our standards (large wikis take 300-4... [14:40:38] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Jclark-ctr) drive changed [14:50:07] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10jcrespo) ` root@db1123:~$ megacli -PDRbld -ShowProg -PhysDrv [32:9] -aALL Rebuild Progress on Device at Enclosure 32, Slot 9 Completed 12% in 9 Minutes. Exit Code: 0... [15:06:55] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) I believe I already defined the racking proposal at T235659 [15:08:44] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) >>! In T241336#5760280, @jcrespo wrote: > @Papaul sorry he will be away, I don't have a clear solution, but I believe any other place on the same row, but not the sam... [15:23:22] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10jcrespo) @Marostegui I've noticed this server has a strip size of 64K. I think as a rule we should audit the RAID configuration on first setup, as it cannot be changed without destroying all data on disks... [15:25:39] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10jcrespo) [17:26:05] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) @jcrespo thanks [17:27:58] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) @jcrespo we will have to move es2024: Row A rack A4 too since A4 is also a 10G rack. Thanks. [17:52:57] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10wiki_willy) [17:53:21] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10wiki_willy) a:03Jclark-ctr [20:11:08] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) >>! In T241336#5761081, @Papaul wrote: > @jcrespo we will have to move es2024: Row A rack A4 too since A4 is also a 10G rack. > > Thanks. Anywhere within row A is... [20:12:06] 10DBA, 10Operations, 10ops-eqiad: eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) Partitioning recipe was also already defined in puppet some days ago so we are good on that front too. Thanks guys [21:04:59] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Reedy) [21:05:01] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Reedy) [21:05:08] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Reedy) No rush, mostly here to keep the schema in sync with changes in the repo [22:06:10] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Paladox) [22:12:07] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10RhinosF1) [22:39:51] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Reedy) [23:12:25] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Reedy)