[02:33:08] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945103 (10alanajjar) @Marostegui Here? [02:34:03] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945104 (10Cyberpower678) He said Monday. [02:36:31] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945105 (10alanajjar) @Cyberpower678 My time zone UTC+02:00 So here is Monday :) @@Marostegui can you write your time zone in your profile? [02:38:37] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945106 (10Cyberpower678) He's a server admin, so more than likely EST which puts us at 21:37. Besides in your time zone no server admin is awake to maintain any site. :) [07:42:02] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945145 (10Marostegui) I am around now [07:50:47] 10DBA, 10Operations, 10ops-codfw: db2039 disk in predictive failure - https://phabricator.wikimedia.org/T186479#3945149 (10Marostegui) [07:51:39] 10DBA, 10Operations, 10ops-codfw: db2039 disk in predictive failure - https://phabricator.wikimedia.org/T186479#3945161 (10Marostegui) p:05Triage>03Normal [09:33:02] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3945362 (10Marostegui) I am fixing now inconsistencies on user_newtalk on the master (db1052), it will take quite sometime. It is the last host to fix. [09:39:55] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T186321#3945379 (10Marostegui) For s6 I would suggest db1088. It is a powerful host, but the only non-powerful host available is db1063 which already ha... [10:46:54] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T186321#3945541 (10Marostegui) [10:52:05] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3945562 (10Marostegui) I have fixed quite lots of rows on db1052 user_newtalk. The problem is that this table doesn't have a PK (T146585) so there are lots of duplicate rows, ie: ``` +---------+---------... [10:58:57] 10DBA: Rebuild user_newtalk on db1052 - https://phabricator.wikimedia.org/T186503#3945590 (10Marostegui) p:05Triage>03Normal [11:01:51] 10DBA: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#3945612 (10Marostegui) [11:01:54] 10DBA: Rebuild user_newtalk on db1052 - https://phabricator.wikimedia.org/T186503#3945611 (10Marostegui) [11:02:08] 10DBA: Rebuild user_newtalk on db1052 - https://phabricator.wikimedia.org/T186503#3945590 (10Marostegui) [11:12:10] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3945657 (10Marostegui) Next table: text [11:12:44] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T186321#3945659 (10Marostegui) [11:14:09] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T186321#3940891 (10Marostegui) For s7 it should probably be db1069 (only non powerful host doesn't need to be decommissioned on the next batch, currentl... [12:16:00] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3945789 (10Marostegui) p:05Triage>03Normal a:03Marostegui [12:22:26] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3945804 (10Marostegui) [13:28:00] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945897 (10MarcoAurelio) @Marostegui - still around? If @Cyberpower678 is still not around I can babysit this if that's okay? [13:28:43] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945898 (10Marostegui) Go for it [13:29:14] marostegui: esperaré un momento por si acaso aparece cyber [13:29:20] no quiero broncas [13:29:25] hehe vale :) [13:29:30] people fight for this things :S [13:29:37] :| [13:29:41] why? [13:31:11] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945900 (10Cyberpower678) I’m around. [13:31:28] marostegui: ni idea [13:31:35] pero ya está ahí [13:32:08] pues adelante :) [13:33:27] has just started [13:34:03] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945915 (10Marostegui) When you start, can you paste the progress URL? Thanks! [13:34:12] https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Qehath [13:34:55] I guess local rename jobs would be a lot of UPDATE commands [13:35:24] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945916 (10Cyberpower678) https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Qehath [13:51:26] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3945935 (10Cyberpower678) The dangerous part is finished. [14:38:55] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3923718 (10Ks-M9) I'm also looking this process (at the link given by @Cyberpower678) and it seems to be failed in Meta: the rename progress indicated that "failed" (and not... [14:39:55] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946072 (10Ks-M9) p:05Triage>03Normal [14:40:13] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946073 (10Cyberpower678) p:05Normal>03Unbreak! [14:40:17] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946075 (10MarcoAurelio) Wait at least 3 hours since the failure because jobs try to restart themselves. [14:40:30] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946076 (10Cyberpower678) This is why we have sysadmins folks. :p [14:40:36] marostegui: se rompió el carro [14:40:38] xD [14:40:55] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946079 (10Cyberpower678) p:05Unbreak!>03High [14:43:34] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946086 (10MarcoAurelio) https://wikitech.wikimedia.org/wiki/Stuck_global_renames [14:50:33] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946095 (10alanajjar) As @MarcoAurelio said, we should wait at least 3 hours. It's happened with me few times, and usually after 1 hours it's fixed spontaneously. [14:51:07] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946096 (10MarcoAurelio) [14:52:02] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946101 (10MarcoAurelio) [14:56:17] jynus: you around? [15:22:10] Hauskatze: Yeah, I saw that [15:22:23] marostegui: it's working again [15:22:31] ah nice [15:22:41] the job restarted itself apparently [15:23:14] cool!! [15:23:36] going out for merienda [15:24:59] xdddd [15:37:49] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3946346 (10Marostegui) [16:12:28] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3946491 (10Marostegui) I have renamed wikidata tables on s5 on all the replicas in both, eqiad and codfw. Tomorrow I will drop them if no errors arise. [16:16:10] 10DBA, 10Wikimedia-Site-requests: Global rename of Dick Laurent → Qehath: supervision needed - https://phabricator.wikimedia.org/T185719#3946494 (10Ks-M9) 05Open>03Resolved Now the process is done after waiting for some hours. All CentralAuth acounts now have the new name: https://meta.wikimedia.org/wiki/S... [16:23:49] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2039 - https://phabricator.wikimedia.org/T186533#3946510 (10Marostegui) p:05Triage>03Normal [16:24:27] 10DBA, 10Operations, 10ops-codfw: db2039 disk in predictive failure - https://phabricator.wikimedia.org/T186479#3946517 (10Marostegui) [16:24:29] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2039 - https://phabricator.wikimedia.org/T186533#3946506 (10Marostegui) [16:30:16] 10DBA, 10Operations, 10ops-codfw: db2039 disk in predictive failure - https://phabricator.wikimedia.org/T186479#3946531 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete. [16:31:44] 10DBA, 10Operations, 10ops-codfw: db2039 disk in predictive failure - https://phabricator.wikimedia.org/T186479#3946535 (10Marostegui) Thanks @Papaul - that was fast! Will close once it is completed ``` logicaldrive 1 (3.3 TB, RAID 1+0, Recovering, 2% complete) physicaldrive 1I:1:1 (port 1I:box... [16:32:16] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2039 - https://phabricator.wikimedia.org/T186533#3946546 (10Marostegui) Thanks @Papaul - that was fast! Will close once it is completed ``` logicaldrive 1 (3.3 TB, RAID 1+0, Recovering, 2% complete) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB,... [16:44:01] 10DBA, 10Data-Services, 10Tool-Global-user-contributions, 10Toolforge, 10cloud-services-team (Kanban): Database error: Unable to connect to s7.web.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T182916#3946594 (10jcrespo) @Krinkle- I am surprised we are not more in line here; let me compare wit... [17:28:19] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3925325 (10RobH) Why did we call this tendril2001 in codfw, but db1115 in eqiad? [17:30:49] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3946714 (10Marostegui) See discussion at T186123 and starting at: T185788#3940445 (basically when I created this ticket I didn't know it was being decided t... [17:32:03] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3946723 (10Marostegui)