[03:37:52] 10DBA: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Sotiale) [04:55:15] 10DBA: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Marostegui) Thanks for the heads up Could you post the UTC time as well? Thanks! [05:29:54] 10DBA: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Sotiale) >>! In T238512#5670164, @Marostegui wrote: > Thanks for the heads up > Could you post the UTC time as well? > > Thanks! Ah! It's 14:00 (2 p.m.) by UTC. Thx! [05:42:31] 10DBA, 10Operations: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Marostegui) p:05Triage→03Normal Thank you! Normally there is no need to tag #dba for such operations, but given how big these two renames are, I am also going to tag #Operations and ping a few people from Wi... [05:42:58] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Operations: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Marostegui) [05:52:27] 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Marostegui) The user @Dzahn suggests to use I believe it is `phstats`, which only has access... [05:53:05] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [05:55:57] 10DBA, 10Operations, 10User-notice: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) Thank you both for pointing it out and for fixing it! :) [05:57:52] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10Marostegui) #2 has the proxies, databases and CNAMEs ready (T190572#5413180). The hosts are read-only, so it wou... [06:12:13] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0), 10Patch-For-Review: Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Marostegui) >>! In T238370#5668442, @Andrew wrote: >>>! In T238370#5667323, @Marostegui wrote: >> labtestwiki lives somewhere within cloud... [06:17:25] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Operations: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Sotiale) Yes, I also remember that this is the first time such a big rename has happened. You mean, you want me to deal with another account after one accou... [06:24:30] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Operations: Related to scheduled renaming - https://phabricator.wikimedia.org/T238512 (10Marostegui) >>! In T238512#5670210, @Sotiale wrote: > Yes, I also remember that this is the first time such a big rename has happened. > > You mean, you wan... [06:45:28] 10DBA: Investigate possible memory leak on db1115 - https://phabricator.wikimedia.org/T231769 (10Marostegui) MySQL upgraded and restarted. [07:06:31] 10DBA, 10MediaWiki-Categories, 10Wikimedia-General-or-Unknown: Inconsistencies in Database dewiki.p - https://phabricator.wikimedia.org/T238500 (10doctaxon) @Bugreporter: IMHO this task is not a duplicate of T228585, because the problem/the bug is not mentioned there. [09:07:51] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Marostegui) @Anomie you can create the tables whenever you like, the replication filters are in place. Let me know though when they are created so I can double c... [09:08:48] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [09:09:08] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [09:14:29] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) [09:15:52] 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui) [09:58:49] 10DBA, 10Data-Services, 10Operations, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10jhsoby) [09:59:49] 10DBA, 10Data-Services, 10Operations, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10Marostegui) p:05Triage→03Normal Let us know when the database is created so we can sanitize it before sending it to t... [10:21:17] 10DBA: Decommission db2061.codfw.wmnet - https://phabricator.wikimedia.org/T238526 (10Marostegui) [10:21:29] 10DBA: Decommission db2061.codfw.wmnet - https://phabricator.wikimedia.org/T238526 (10Marostegui) p:05Triage→03Normal [10:23:23] 10DBA: Decommission db2061.codfw.wmnet - https://phabricator.wikimedia.org/T238526 (10Marostegui) [10:23:52] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [10:25:12] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [10:26:39] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10Urbanecm) a:05Urbanecm→03None Unassigning, there's nothing I can do here :-). [10:29:56] going to upgrade labsdb to 10.1.43 during these days [10:29:59] labsdb1012 is done [10:30:13] I am going to try to upgrade the whole s7 chain to 10.1.43 before the failover next week [10:30:20] so that includes labs and sanitarium [11:21:43] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for minwiktionary - https://phabricator.wikimedia.org/T238522 (10jhsoby) Right, sorry – forgot to remove the defaults when creating the task. :-) [11:46:38] 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Aklapper) That's also my understanding of the minimum "core databases" I'd need. So let's go... [11:47:03] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Operations: Scheduled renaming of two Wikimedia user accounts - https://phabricator.wikimedia.org/T238512 (10Aklapper) [13:06:55] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10Operations: Scheduled renaming of two Wikimedia user accounts - https://phabricator.wikimedia.org/T238512 (10Ladsgroup) As I said in the global renamers mailing list, the only thing I'm worried about is FlaggedRevs and AbuseFilter tables that mig... [14:17:07] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) [14:18:00] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) This is how the table looks like after the ALTER: ` Table: wbt_text_in_lang Create Table: CREATE TABLE `wbt_text... [14:18:42] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [14:18:46] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [14:23:12] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) s8 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1109 [] db1104 [] db1101 [] db1099 [] db1092 []... [14:23:23] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) s8 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1109 [] db110... [14:23:30] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) s8 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db1126 [] db1124 [] db1116... [14:57:36] marostegui, can you help me understand what's happening with https://phabricator.wikimedia.org/T237509? Did I need to run maintain-views but /not/ remove that table? Or write some more selective query in the yaml file? [14:59:26] andrewbogott: meeting, I will help you in 1h or so :) [15:01:25] cool, thanks [16:15:13] andrewbogott: so basically, that table was removed from everywhere (it was empty) but from centralauth cause that's where it lives [16:15:26] By running --clean (or whatever option it was) that also removed the view from that valid database [16:15:46] marostegui: We're thinking that if I replace it in maintain_views.yaml the script will just ignore it where it's missing anyway [16:15:48] So I guess you'd need to tackle that exception from the script, but I don't know enough about its internal usage to see how to do that [16:15:51] does that sound right to you? [16:16:01] andrewbogott: replace it with what? [16:16:21] just revert https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/550888/ [16:17:21] andrewbogott: Will that fail if run with --replace as it tries to create a view with a table that doesn't exist? [16:17:53] I think it'll just skip it [16:17:53] https://github.com/wikimedia/puppet/blob/production/modules/profile/files/labs/db/views/maintain-views.py#L120 [16:18:14] ah, interesting [16:18:29] I would make a note on the source file that create the views, to reference this task [16:18:30] for the future [16:18:41] because it is weird to leave a view that only applies to centralauth [16:18:47] yeah, agreed [16:19:08] but, ok, if reverting that patch and re-running maintain_views doesn't seem obviously wrong to you I will try that :) [16:19:16] Assuming you don't have anything else in the works that might conflict [16:19:33] yep, not working on the replicas atm [16:19:51] great, will start on that in a bit [16:32:56] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [16:49:35] Hi, if anybody knows something about: I only have minimal knowledge about access to replica. I use the username u4802 and a password to get access to replica. Is there a manual about this logon process? [16:50:14] about DB connection? [16:50:51] yes [16:51:52] https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas [17:00:20] thanks, can I get one more client user name and password. Several time a day I have to do some longer lasting (about ten minutes) SQL queries parallely, because there are launched by various scripts. [17:04:22] doctaxon: please talk to wikimedia cloud team at #wikimedia-cloud [17:26:41] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Anomie) [17:26:47] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Anomie) >>! In T238370#5670422, @Marostegui wrote: > @Anomie you can create the tables whenever you like, the replication filters are in place. Let me know thoug... [17:27:03] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) considering we no have T237259 and the fact that the sooner we install the new certificate the more chance difficult services i.e. mysql servers will natural... [17:29:52] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Marostegui) Thanks @Anomie, filters are working fine! ` root@labsdb1009:/srv/sqldata# find . -name oauth2_access_tokens* root@labsdb1009:/srv/sqldata# ` [17:31:31] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Marostegui) [19:04:16] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10Ottomata) [19:04:35] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10Ottomata) [19:06:57] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10Ottomata) [19:07:48] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10Ottomata) [19:58:21] 10Blocked-on-schema-change, 10DBA, 10CPT Initiatives (OAuth 2.0): Apply schema changes for OAuth 2.0 - https://phabricator.wikimedia.org/T238370 (10Andrew) I applied all those changes to labtestwiki [23:15:12] 10DBA, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Read access for phabricator-admins (aklapper) to Phabricator production database to run SELECT queries - https://phabricator.wikimedia.org/T238425 (10Dzahn) @Aklapper Given that Phabricator uses multiple databases, do you just need the "phabri...