[00:24:33] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Wilfredor) It can be done anytime, Im waiting
[06:59:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui)
[06:59:36] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) I have rebooted db2085 without debug option on kernel as part of (T216273) and I have taken the opportunity to upgrade its kernel too.
[08:32:22] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui)
[08:38:00] <wikibugs>	 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10Marostegui) I think we can close this, we already have the tasks in place: T216142 T216138 T216137 T214069 T214066
[09:02:20] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui)
[09:04:50] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] dbstore1002 [] dbstore1001 [] db...
[09:05:23] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui)
[09:32:44] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui)
[10:50:43] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui)
[13:52:02] <wikibugs>	 10DBA: Document clearly the mariadb backup and recovery setup - https://phabricator.wikimedia.org/T205626 (10jcrespo)
[14:22:31] <wikibugs>	 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui)
[14:39:59] <hauskatze>	 marostegui: hi - global rename possible?
[14:40:48] <marostegui>	 hauskatze: sure, is it https://phabricator.wikimedia.org/T215107 ?
[14:41:03] <hauskatze>	 sí
[14:41:09] <marostegui>	 go for it!
[14:41:16] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) I am free to do it today if @Marostegui or other DBA is willing to have a look while the rename is ongoing.
[14:41:19] <hauskatze>	 need links, etc?
[14:41:25] <marostegui>	 can you paste the url progress?
[14:41:27] <marostegui>	 yeah, the progress one
[14:41:33] <hauskatze>	 once I get it, sure
[14:41:40] <marostegui>	 gracias!
[14:41:43] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Marostegui) Go for it!
[14:42:03] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) a:03MarcoAurelio
[14:44:59] <hauskatze>	 marostegui: https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Wilfredor
[14:45:20] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) Rename is ongoing: https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Wilfredor
[14:45:40] <marostegui>	 thanks!
[14:45:50] <marostegui>	 there we go commons
[14:46:04] <hauskatze>	 yup
[14:46:25] <hauskatze>	 poor dbs
[14:46:29] <marostegui>	  yeah, I can see the update on the master
[14:46:38] <marostegui>	 we will see if it makes codfw lag
[14:46:53] <hauskatze>	 we ain't using codfw for nothing atm right?
[14:46:58] <marostegui>	 yep
[14:46:59] <hauskatze>	 it's just the secondary dc
[14:47:00] <jynus>	 wouldn't, in theory, renames be an instant action "soon"?
[14:47:03] <marostegui>	 yes
[14:47:09] <marostegui>	 jynus: so far the update has been running for a minute
[14:47:12] <marostegui>	 :)
[14:47:15] <hauskatze>	 jynus: from theory to practice...
[14:47:24] <jynus>	 even for millions of revisions?
[14:47:26] <hauskatze>	 still running on commons
[14:47:35] <marostegui>	 yep, I can see the update
[14:48:24] <marostegui>	 done on the master
[14:49:49] <hauskatze>	 still in progress there apparently?
[14:51:31] <marostegui>	 yep, apparently, I don't know what it does behind the scenes, the master looks done from the CLI point of view
[14:53:56] <marostegui>	 maybe it failed?
[14:54:06] <marostegui>	 On the user table I still see the old name
[14:54:59] <marostegui>	 yep, it timeout as per logstash
[14:56:06] <hauskatze>	 great...
[14:56:22] <hauskatze>	 well, the script docs says it restarts each two/three hours?
[14:56:37] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Marostegui) It timed out on Commonswiki: https://logstash.wikimedia.org/goto/34de73560ce6692f0012e846f7a4de0c Maybe @Legoktm can help to unbl...
[14:56:45] <marostegui>	 Yeah, I pinged legoktm on the task too, just in case he can run his magic
[14:56:55] <hauskatze>	 now I feel bad
[14:57:07] <marostegui>	 Why?
[15:15:34] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) According to https://wikitech.wikimedia.org/wiki/Stuck_global_renames the system should try to restart failed jobs several time...
[15:20:16] <hauskatze>	 marostegui: because of the failed rename. I understand it is not my fault that jobs time out but...
[15:20:28] <hauskatze>	 has the job attempted to restart again?
[15:21:35] <marostegui>	 hauskatze: no idea :)
[15:34:28] <jynus>	 so heads up, logical backups are now on /srv/backups/dumps
[15:34:53] <jynus>	 I will do some testing before the weekly process runs
[15:34:59] <marostegui>	 great!
[16:30:59] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10JAllemandou) Hi @Isaac, I have generated some parquet data here `/user/joal/wmf/data/wmf/wikidata/item_page_link/20190204`  with th...
[17:41:18] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) @Tgr has helped us in the past, too, with stuck global renames.  @Marostegui (or anyone with access) could you see the wikitech...
[17:44:20] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Marostegui) There have been no retries from what I can see on: https://logstash.wikimedia.org/goto/65afdb88fef30982130c53e40a644b06
[17:45:10] <hauskatze>	 marostegui: according to the Wikitech guide, it should be restarting soon (3 hours passed already), could you check?
[17:53:51] <hauskatze>	 It looks we may need to use that maintenance script to unblock the rename
[17:56:35] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10MarcoAurelio) Just in case it is needed:  `name=On mwmaint1001 mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=co...
[18:00:25] <hauskatze>	 bbl
[19:59:50] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Дагиров Умар → Takhirgeran Umar: supervision needed - https://phabricator.wikimedia.org/T216444 (10Nihlus)
[20:07:23] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of Дагиров Умар → Takhirgeran Umar: supervision needed - https://phabricator.wikimedia.org/T216444 (10Marostegui) This should wait until T215107 is unblocked and resolved
[20:10:24] <hauskatze>	 marostegui: another global rename heh
[22:23:53] <wikibugs>	 10DBA, 10CheckUser, 10Patch-For-Review: Create index for cu_agents in cu_changes table - https://phabricator.wikimedia.org/T147894 (10Huji) @jcrespo The index you created in T212092#4934152 is exactly the one proposed in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CheckUser/+/414120/1/cu_changes....
[23:27:00] <wikibugs>	 10DBA, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Tgr) Large renames should be easy once actor migration (T167246) is finished, right? I'd just block them for now.  Job started on Commons at...