[05:15:10] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3641737 (10Marostegui) [05:22:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3641739 (10Marostegui) [05:45:23] 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3641756 (10Marostegui) 05Open>03Resolved a:03Marostegui In order to get this over with - I have run myself: ``` maintain-views --databases hiwikivoyage... [05:57:27] 10DBA: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3641763 (10Marostegui) [05:57:46] 10DBA: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3641776 (10Marostegui) p:05Triage>03Normal [06:00:35] 10DBA: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3641778 (10Marostegui) [06:18:03] 10DBA, 10Operations: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#3641783 (10Marostegui) [06:18:32] 10DBA, 10Operations: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#2714228 (10Marostegui) db1035 scheduled for decommission: T176931 [13:19:02] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3642638 (10Marostegui) [13:19:46] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3564203 (10Marostegui) s4 is totally done. Only excluded: dbstore1001 and dbstore1002 because they are tokudb [13:23:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3642651 (10Marostegui) [13:25:02] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3564203 (10Marostegui) [13:35:11] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3642702 (10Marostegui) [14:39:32] 10DBA, 10Patch-For-Review: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3643066 (10Marostegui) [14:54:45] 10DBA, 10Operations, 10ops-codfw: db2044 HW RAID failure - https://phabricator.wikimedia.org/T174764#3643107 (10Papaul) After reviewing all the logs , HP decide to send a new mainboard because the controller is on the mainboard. The tech will be onsite to perform the replacement on Monday 2nd between 9am and... [14:56:21] 10DBA, 10Operations, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3643109 (10Marostegui) a:05Marostegui>03Cmjohnson db1035 is ready to be decommissioned by @Cmjohnson All the configuration references have been removed, it has been set to spare, alerts disabled and MySQL i... [14:58:21] 10DBA, 10Operations, 10ops-codfw: db2044 HW RAID failure - https://phabricator.wikimedia.org/T174764#3643117 (10Marostegui) >>! In T174764#3643107, @Papaul wrote: > After reviewing all the logs , HP decide to send a new mainboard because the controller is on the mainboard. The tech will be onsite to perform... [15:04:43] 10DBA, 10Operations, 10ops-codfw: db2044 HW RAID failure - https://phabricator.wikimedia.org/T174764#3643143 (10Papaul) @Marostegui yes I will make sure PXE is disable. [15:05:06] 10DBA, 10Operations, 10ops-codfw: db2044 HW RAID failure - https://phabricator.wikimedia.org/T174764#3643144 (10Marostegui) Thank you! [15:50:42] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643401 (10alanajjar) [15:51:15] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643413 (10alanajjar) [15:51:50] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643417 (10Marostegui) p:05Triage>03Normal When would you like to do this rename? In which timezone are you? [15:52:19] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643420 (10alanajjar) p:05Normal>03Triage [15:52:46] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643401 (10alanajjar) @Marostegui if you available, we can do it now? [15:53:03] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643427 (10Marostegui) >>! In T176985#3643420, @alanajjar wrote: > @Marostegui if you available, we can do it now? Sure, give me 5 minutes to get ready [15:53:44] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643429 (10alanajjar) @Marostegui okay, when you being ready ping me here :) [15:55:56] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643447 (10Marostegui) >>! In T176985#3643427, @Marostegui wrote: >>>! In T176985#3643420, @alanajjar wrote: >> @Marostegui if you available, we can do it now? > >... [16:01:17] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643494 (10alanajjar) @Marostegui [[https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Rua|the progress]] [16:01:59] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643500 (10Marostegui) Thanks! [16:07:26] https://twitter.com/mysqlborat/status/604026894133170180 [16:07:52] haha [16:08:10] that could have been a jynus kind ofjoke [16:08:51] I suspect he might in fact be the man behind that twitter account [16:10:04] probably [16:10:32] that and this account too: https://twitter.com/TrumpDBA [16:11:47] I am definitely not DBA Borat not TrumpDBA [16:12:21] yeah yeah sure [16:17:44] Proper upgrade path to #MariaDB 10 is #MySQL 5.5, #Drizzle 6, #PostgreSQL 7, #Windows 8, #Oracle 9i. [16:20:58] lol [16:28:43] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643613 (10Marostegui) @alanajjar any way to check the progress of the "in progress" status? :-) It's been there for a while now [16:31:20] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, and 2 others: Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" - https://phabricator.wikimedia.org/T176055#3643616 (10Bawolff) 05Open>03Resolved a:03Baw... [16:40:02] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643682 (10alanajjar) @Marostegui if you see [[https://en.wiktionary.org/w/index.php?title=Special%3ACentralAuth&target=Rua|the Central Auth]] of the new name (Rua) y... [16:41:26] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643684 (10Marostegui) >>! In T176985#3643682, @alanajjar wrote: > @Marostegui if you see [[https://en.wiktionary.org/w/index.php?title=Special%3ACentralAuth&target=R... [16:47:27] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643701 (10alanajjar) @Marostegui I think there's a problem here! If you see [[https://en.wiktionary.org/w/index.php?limit=50&title=Special%3AContributions&contribs=... [16:49:06] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643712 (10Marostegui) >>! In T176985#3643701, @alanajjar wrote: > @Marostegui I think there's a problem here! > > If you see [[https://en.wiktionary.org/w/index.php... [16:51:31] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643720 (10Marostegui) @MarcoAurelio could this be something related to T173419? This rename got stuck in enwiktionary (where most of the edits are) [16:54:28] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643730 (10alanajjar) >>! In T176985#3643720, @Marostegui wrote: > @MarcoAurelio could this be something related to T173419? > This rename got stuck in enwiktionary (... [17:13:18] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643804 (10MarcoAurelio) @Marostegui Please run mwscript showJobs.php --wiki=enwiktionary on terbium so we can know if there are queued jobs there. [17:13:54] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643805 (10Marostegui) >>! In T176985#3643804, @MarcoAurelio wrote: > @Marostegui Please run > > mwscript showJobs.php --wiki=enwiktionary > > on terbium so we ca... [17:15:29] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643809 (10Marostegui) I saw that this query timed out on enwiktionary - on the logs: ``` SELECT log_timestamp FROM `logging` WHERE log_user_text = 'CodeCat' AND... [17:22:37] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643848 (10MarcoAurelio) (conversation being continued on IRC) [17:23:43] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643849 (10Marostegui) brief update: @MarcoAurelio and myself have been chatting on IRC ``` ˜/tabbycat 19:16> marostegui: hi ˜/tabbycat 19:16> marostegui: with --grou... [17:48:05] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3643932 (10Marostegui) After more chatting and checks @marcoaurelio suggested: ``` mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikti... [18:13:47] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3644046 (10MarcoAurelio) However renames happening after en.wiktionary are leaving unnatached local accounts. @Legoktm suggest running after the rename has finished:... [18:38:08] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3644119 (10Marostegui) The command to attach the accounts didn't fix the issue: ``` root@terbium:~# cat /tmp/rua.txt Rua root@terbium:~# mwscript extensions/CentralA... [19:24:38] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3644406 (10Marostegui) ``` ˜/legoktm 21:23> the good news is that nothing is wrong with the account, just how it is displayed on Special:CentralAuth :) ˜/legoktm 21:2... [19:31:42] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3644420 (10Marostegui) 05Open>03Resolved a:03Legoktm Everything is fixed after running the UPDATE below ``` ˜/legoktm 21:28> !log mysql:wikiadmin@db1062 [centr... [20:18:42] 10DBA, 10Gerrit, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Next): Gerrit is failing to start gerrit-ssh on gerrit2001 - https://phabricator.wikimedia.org/T176532#3644612 (10Paladox) Adding DBA as we need the firewall to allow connection from m2-master.codfw.wmnet. Since the eqiad has... [20:22:50] 10DBA, 10Gerrit, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Next): Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working - https://phabricator.wikimedia.org/T176532#3644618 (10Paladox) [23:19:31] 10DBA, 10Community-Tech-Sprint, 10MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is pai... - https://phabricator.wikimedia.org/T175962#3645037 [23:19:36] 10DBA, 10Community-Tech, 10cloud-services-team, 10Security: create production ip_changes table for RangeContributions - https://phabricator.wikimedia.org/T173891#3645038 (10MaxSem) [23:28:12] 10DBA, 10Community-Tech, 10cloud-services-team, 10Security: Create core ip_changes view for replicas - https://phabricator.wikimedia.org/T173891#3645059 (10MusikAnimal) [23:29:25] 10DBA, 10Community-Tech, 10cloud-services-team, 10Security: Create core ip_changes view for replicas - https://phabricator.wikimedia.org/T173891#3543452 (10MusikAnimal) The table is now live and populated on all wikis, so I've repurposed this ticket for creating the replicas view. Hope that is OK. Any upda...