[01:43:10] committed some of my fiddling earlier to get beta back up at https://gerrit.wikimedia.org/r/c/operations/puppet/+/560867 [01:43:35] it's cherry-picked, don't do anything that'll cause nginx to be restarted (e.g. instance restart) without it [01:44:09] also fixed puppet on deployment-puppetdb02 [02:13:52] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10Tulsi_Bhagat) @Masumrezarock100 I understand your view and that's okay. Everyone has their own perspective to see things. It might seems like I am showing off my rights, but it's nothing... [03:12:25] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10Krenair) 05Open→03Declined beta is not intended for playing with various restricted MW functionality, it is for developers to test changes before they go out to production [03:24:40] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10Tulsi_Bhagat) @Krenair I appreciate it. Thank you for clarifying. I was like I can test minorly, but It's like Beta Cluster is more a developer wiki. [04:45:25] 10Beta-Cluster-Infrastructure, 10User-DannyS712, 10Wikimedia-maintenance-script-run: Run emptyUserGroup on the beta cluster - https://phabricator.wikimedia.org/T241504 (10DannyS712) [07:57:11] 10Beta-Cluster-Infrastructure, 10User-DannyS712, 10Wikimedia-maintenance-script-run: Run emptyUserGroup on the beta cluster - https://phabricator.wikimedia.org/T241504 (10Bugreporter) See also {T105570} [08:09:05] (03PS1) 10Zoranzoki21: Archive RecentActivityFeed extension [integration/config] - 10https://gerrit.wikimedia.org/r/560874 (https://phabricator.wikimedia.org/T241032) [08:10:04] (03PS2) 10Zoranzoki21: Archive RecentActivityFeed extension [integration/config] - 10https://gerrit.wikimedia.org/r/560874 (https://phabricator.wikimedia.org/T241032) [10:35:07] (03PS1) 10Siebrand: Edit Project Config [extensions/GoogleAdSense] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/560887 [10:40:36] (03PS1) 10Zoranzoki21: Edit Project Config [extensions/RandomArea] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/560888 [10:41:42] (03Abandoned) 10Zoranzoki21: Edit Project Config [extensions/RandomArea] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/560888 (owner: 10Zoranzoki21) [10:47:54] (03CR) 10Zoranzoki21: [C: 04-1] "?" [extensions/GoogleAdSense] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/560887 (owner: 10Siebrand) [11:00:40] (03CR) 10Siebrand: [C: 03+1] Archive RecentActivityFeed extension [integration/config] - 10https://gerrit.wikimedia.org/r/560874 (https://phabricator.wikimedia.org/T241032) (owner: 10Zoranzoki21) [12:04:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) I performed a rename yesterday after various instances and services restarts, and other maintenance (cfr. T241462). It was a... [12:04:55] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) I performed a rename yesterday after various instances and services restarts, and other mai... [12:16:59] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) Nah, it does not work: ` maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/m... [12:20:11] 10Beta-Cluster-Infrastructure, 10User-DannyS712, 10User-MarcoAurelio, 10Wikimedia-maintenance-script-run: Run emptyUserGroup on the beta cluster - https://phabricator.wikimedia.org/T241504 (10MarcoAurelio) a:03MarcoAurelio [12:22:21] (03Abandoned) 10Siebrand: Edit Project Config [extensions/GoogleAdSense] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/560887 (owner: 10Siebrand) [13:13:19] !log maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php petitiondata # T241504 [13:13:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:13:23] T241504: Run emptyUserGroup on the beta cluster - https://phabricator.wikimedia.org/T241504 [13:19:22] Urbanecm: hi? [13:19:35] hi hauskatze, what's up? [13:20:12] Urbanecm: do you by chance know who are the DB_MASTER and DB_REPLICA for deployment-prep? [13:20:25] s/who/which [13:21:23] also, mwscript sql.php deploymentwiki fatals [13:21:29] hauskatze: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/db-labs.php should answer your question [13:21:40] deployment-db05 is master, deployment-db06 slave [13:21:43] I knew there should be someplace :) [13:21:52] aha, okay, so my suspicions were right [13:22:11] Urbanecm: so I logged in to deployment-db05 via ssh and tried to sql there, but it won't let me [13:22:18] says "sql" doesn't exist [13:22:19] lol [13:22:59] Urbanecm: context: https://phabricator.wikimedia.org/T241294#5765164 [13:23:01] interesting...what do i do differently? https://www.irccloud.com/pastebin/cpcqZ5VR/ [13:23:47] on db05/06 I mean [13:23:59] sql deploymentwiki; on deploy01 worked fine [13:24:27] hauskatze: instead of logging into deployment-db05, you can do mysql -h deployment-db05 -u wikiadmin -p from deploy01 [13:24:48] pwd is in /srv/mediawiki-staging/private/PrivateSettings.php [13:25:04] aha, it was the -p I didn't know [13:25:08] I'll see if I can do that [13:25:18] I need to know if there's data discrepancy [13:25:32] * hauskatze goes back to deploy01 [13:29:50] okay, preparing to see [13:34:10] Hmm, apparently centralauth.renameuser_status is both the same on db05 and db06 if I did things right [13:34:54] hauskatze: btw, I wouldn't trust showJobs.php, it doesn't seem to work with kafka-based job queue (it shows zero at prod as well, which is never true) [13:35:13] indeed, I suspected something was wrong with that script [13:37:41] Well, I'm not sure what to do now. [13:38:00] I'm sure there's a reason why fixStuckGlobalRename.php do not detect those renames in progress [13:38:30] lunch, bbl [13:44:44] (03CR) 10Umherirrender: [C: 03+1] "Needs manual rebasing before merging" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) (owner: 10Daimona Eaytoy) [14:20:59] !log urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=deploymentwiki 'DannyS712 tes' 'DannyS712 test' (T241294) [14:21:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:21:04] T241294: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 [14:23:50] !log urbanecm@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=arwiki --logwiki=deploymentwiki 'Riley Huntley' '~riley' (T241294) [14:23:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:24:36] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10Urbanecm) @MarcoAurelio Found the issue. fixStuckGlobalRename.php requires **spaces**, the MediaWiki doct... [14:27:05] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10Urbanecm) [Added](https://wikitech.wikimedia.org/w/index.php?title=Stuck_global_renames&diff=1849244&oldi... [14:28:06] Urbanecm: so not double quotes needed? [14:28:15] '' "' but ' ' ? [14:28:47] hauskatze: that, and spaces, not underscores [14:28:58] I didn't put underscores [14:29:03] so has to be the quotes [14:29:46] hauskatze: https://phabricator.wikimedia.org/T241294#5759652 shows you did, but anyway - it's solved :) [14:30:14] Urbanecm: not today : https://phabricator.wikimedia.org/T241294#5765164 :) [14:30:19] but I'm glad it's solved [14:30:39] unless my Terminal is doing strange stuff [14:30:51] there's still DannyS712_test unless I'm totally blind [14:31:20] oh right [14:31:23] sorry [14:31:30] I'm the blind apparently [14:31:42] single quotes and no underscores is the trick [14:32:05] now I wonder if the script should be more smart to work with and without underscores [14:32:30] Probably should - let's fill a task? [14:33:06] It should work fine as long as the name is quoted [14:33:06] I'm checking the jobrunner logs and looks it's shifting slowly [14:33:23] $oldName = User::getCanonicalName( $this->getArg( 0 ) ); [14:33:23] $newName = User::getCanonicalName( $this->getArg( 1 ) ); [14:33:29] Reedy: with " " or ' ' ? [14:33:42] I would normally use "" but both should work [14:33:59] There's one more stuck rename I can test this [14:34:20] I certainly wouldn' expect it to work without quotes [14:34:24] And nor should it [14:35:04] Knowing whether Foo Bar Baz is Foo_Bar -> Baz, Foo -> Bar_Baz or Foo_Bar_Baz -> ? [14:39:05] Urbanecm: looks it's indeed working https://logstash-beta.wmflabs.org/goto/08c401ffdbcf69d2cb546d1ed70ddb69 [14:39:12] great! [14:39:15] the slowness is another question [14:39:31] agreed [14:41:29] any reason why starting a job is logged at INFO level, and finishing it at DEBUG? [14:41:43] maybe to filter them? [14:57:33] Urbanecm: using single quotes for the other rename did the trick indeed [14:58:10] !log maurelio@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'HoseaSeverson6' 'Random spammer username' | T241294 [14:58:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:58:15] T241294: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 [15:08:55] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) There's now no stuck global renames anymore; but the issue with renames taking forever to p... [15:09:39] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) >>! In T241294#5765194, @Urbanecm wrote: > @MarcoAurelio Found the issue. fixStuckGlobalRen... [15:25:15] (03PS10) 10Daimona Eaytoy: Enforce docblock on private methods [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) [15:26:41] (03PS11) 10Daimona Eaytoy: Enforce docblock on private methods [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) [15:29:07] (03PS12) 10Daimona Eaytoy: Enforce docblock on private methods [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) [16:21:57] (03CR) 10Umherirrender: [C: 03+2] Enforce docblock on private methods [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) (owner: 10Daimona Eaytoy) [16:22:32] (03Merged) 10jenkins-bot: Enforce docblock on private methods [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539663 (https://phabricator.wikimedia.org/T233838) (owner: 10Daimona Eaytoy) [16:26:37] 10MediaWiki-Codesniffer: Create optional sniff to require documenting private methods - https://phabricator.wikimedia.org/T233838 (10Daimona) 05Open→03Resolved [16:27:51] (03CR) 10Umherirrender: "From the task description the Thanks extension should also be part of the dependency." [integration/config] - 10https://gerrit.wikimedia.org/r/560807 (https://phabricator.wikimedia.org/T235873) (owner: 10DannyS712) [16:29:55] (03CR) 10Daimona Eaytoy: "> From the task description the Thanks extension should also be part" [integration/config] - 10https://gerrit.wikimedia.org/r/560807 (https://phabricator.wikimedia.org/T235873) (owner: 10DannyS712) [21:59:43] (03CR) 10Legoktm: "Bump? :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/551911 (owner: 10Daimona Eaytoy) [22:42:39] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10bd808) Related: * {T241285} * {T241284} [23:35:13] 10MediaWiki-Codesniffer: Remove tabs in function signature - https://phabricator.wikimedia.org/T241520 (10Umherirrender)