[07:15:07] FYI, I'm continuing to rollout debmonitor clients (each install injects it's package state to the debmonitor db on m2). same pace as yesterday, so if you haven't noticed any db issues yesterday, it will be fine today as well [07:15:41] if there should be any issues, please ping me and I'll stop [07:31:51] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 3 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4284632 (10Addshore) [07:32:59] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 4 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#3549134 (10Addshore) This will be set to take the train on the week of the 25th of June, providing everything goes well on group0 over the coming week. [07:33:07] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 4 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#4284638 (10Addshore) [08:58:17] 10DBA, 10Operations, 10Puppet: Move mariadb_maintenance away from terbium/wasat (mediawiki_maintenance) - https://phabricator.wikimedia.org/T184797#4289982 (10Dzahn) p:05Lowest>03Normal a:03jcrespo [09:41:42] 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#4290161 (10Dzahn) [09:41:51] 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#4290164 (10Dzahn) [10:39:45] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290656 (10jcrespo) [10:39:48] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290669 (10jcrespo) p:05Triage>03Low [10:41:28] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290656 (10jcrespo) [10:46:17] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290741 (10Marostegui) wikidatawiki and commons were already done as there was a schema change that involved logging table and it rebuilt that table, and it was done right after the deletes finished on that table, so we mark those as done. [10:46:32] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290742 (10jcrespo) [10:51:52] 10DBA: Optimize logging table - https://phabricator.wikimedia.org/T197459#4290802 (10jcrespo) [10:54:23] 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Patch-For-Review: Configure Toolforge replica views and dumps for the new MCR tables - https://phabricator.wikimedia.org/T184446#4290820 (10Aklapper) [10:54:33] 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Patch-For-Review: Configure Toolforge replica views and dumps for the new MCR tables - https://phabricator.wikimedia.org/T184446#4290827 (10Aklapper) [11:14:26] 10DBA, 10Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#4290941 (10Marostegui) I have been testing the workaround suggested on https://jira.mariadb.org/browse/MDEV-12012?focusedCommentId=100529&page=com.atlassian.jira.plugin.system.issuetabpanels:com... [11:19:51] 10DBA, 10Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#4290952 (10jcrespo) > why gtid_domain_id is getting updates for domain 0 from those two hosts My guess is that I am not 100% sure all masters were restarted after changing its domain_id. [11:24:36] 10DBA, 10Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#4290987 (10Marostegui) That could explain it. We will see if it stops once we failover s1 master. [11:26:43] 10DBA, 10Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#4291001 (10jcrespo) We could also think of testing strict mode to prevent slave, out-of-band writes (being very careful). [11:33:55] 10DBA, 10Analytics, 10EventBus, 10MediaWiki-Categories, and 5 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#4291098 (10jcrespo) [11:38:53] 10DBA, 10Analytics, 10EventBus, 10MediaWiki-Categories, and 6 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#4291167 (10jcrespo) This was known to me. This points me I should be more vocal about #wikimedia-l... [11:47:08] 10DBA, 10Data-Services, 10Quarry: Cannot reliably get the EXPLAIN for a query on analytics wiki replica cluster - https://phabricator.wikimedia.org/T195836#4291244 (10jcrespo) p:05Triage>03Low So the workaround for now is to make sure one is connected to the same server by doing: ``` SELECT @@GLOBAL.hos... [11:53:34] 10DBA, 10AbuseFilter, 10Patch-For-Review: Move AbuseFilter slow filters data from Logstash to per-filter profiling - https://phabricator.wikimedia.org/T179604#4291323 (10jcrespo) Sorry, #DBA was added, but I don't see clear actionables for us. We are happy to help with all database-related tasks, but we need... [12:13:28] 10DBA, 10Analytics, 10EventBus, 10MediaWiki-Categories, and 6 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#4291360 (10Anomie) 05Open>03Resolved a:03Anomie This looks likely to be resolved now: the ch... [12:16:52] 10DBA, 10AbuseFilter, 10Patch-For-Review: Move AbuseFilter slow filters data from Logstash to per-filter profiling - https://phabricator.wikimedia.org/T179604#4291376 (10Daimona) @jcrespo Thanks, and sorry :-) This patch will need approval since it adds a new table, but it's not ready for review, since this... [12:22:43] 10DBA, 10Phabricator, 10Release-Engineering-Team (Next): Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810#4291386 (10jcrespo) @mmodell To clarify, this is blocked on a decision of what you want to do architecture-wise, and I think the best way to move forward is for us to me... [12:25:05] 10DBA, 10MediaWiki-Database, 10Operations: Preserve InnoDB table auto_increment on restart - https://phabricator.wikimedia.org/T135851#4291395 (10jcrespo) This is now fixed on MySQL 8.0 https://dev.mysql.com/doc/refman/8.0/en/innodb-auto-increment-handling.html#innodb-auto-increment-initialization [12:41:11] 10DBA, 10AbuseFilter, 10Patch-For-Review: Move AbuseFilter slow filters data from Logstash to per-filter profiling - https://phabricator.wikimedia.org/T179604#4291437 (10jcrespo) @Daimona Absolutely no problem, I just wanted to set expectations clear that we were not actively working on this (sometimes misun... [12:44:55] 10DBA, 10Reading List Service, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Update duplicate handling in reading lists API - https://phabricator.wikimedia.org/T184680#4291536 (10Aklapper) p:05Lowest>03Triage [12:45:14] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4291556 (10Aklapper) p:05Lowest>03Triage [12:47:59] 10DBA, 10Analytics, 10EventBus, 10MediaWiki-Categories, and 6 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#4291625 (10jcrespo) Please don't run `count(*)` + `LOCK IN SHARE MODE` on the masters or you will... [12:48:16] 10DBA, 10Reading List Service, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Update duplicate handling in reading lists API - https://phabricator.wikimedia.org/T184680#4291634 (10Aklapper) a:03Tgr [13:01:30] 10DBA, 10AbuseFilter, 10Analytics-Kanban, 10Data-release, and 11 others: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#4291754 (10Aklapper) a:03jcrespo [13:01:34] 10DBA, 10AbuseFilter, 10Analytics-Kanban, 10Data-release, and 12 others: Generate consistent logical database backups in CODFW - https://phabricator.wikimedia.org/T184699#4291756 (10Aklapper) a:03jcrespo [13:01:40] 10DBA, 10AbuseFilter, 10Analytics-Kanban, 10Data-release, and 14 others: Decommission db1011 - https://phabricator.wikimedia.org/T184703#4291755 (10Aklapper) a:03Cmjohnson [13:02:30] 10DBA, 10Patch-For-Review: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#4291791 (10Aklapper) p:05Lowest>03Normal [13:02:32] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1011 - https://phabricator.wikimedia.org/T184703#4291794 (10Aklapper) p:05Lowest>03Normal [13:02:41] 10DBA, 10Operations, 10Goal: Generate consistent logical database backups in CODFW - https://phabricator.wikimedia.org/T184699#4291798 (10Aklapper) p:05Lowest>03Normal [13:02:47] 10DBA, 10Patch-For-Review: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#4291804 (10Aklapper) p:05Lowest>03Normal [13:02:49] 10DBA, 10Patch-For-Review: Failover existing eqiad database backup system to the new codfw database logical backup system - https://phabricator.wikimedia.org/T184697#4291801 (10Aklapper) p:05Lowest>03Normal [13:19:34] 10DBA, 10Analytics, 10EventBus, 10MediaWiki-Categories, and 6 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#4291929 (10jcrespo) I see most issues arose from updates to things like 'CC-BY-SA-4.0', and 'Self-... [13:34:52] 10DBA, 10Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#4292108 (10Marostegui) For what is worth, I have replicated exactly the same thing we have in production to make sure nothing would break along the way with the suggested workaround. That is: m... [13:51:43] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2060 - https://phabricator.wikimedia.org/T184464#4292288 (10Aklapper) [13:52:56] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Security: Decommission db1030 - https://phabricator.wikimedia.org/T184397#4292319 (10Reedy) [13:53:34] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1030 - https://phabricator.wikimedia.org/T184397#4292326 (10Reedy) [13:54:27] 10DBA, 10Analytics-Kanban, 10Security: db1011 possibly faulty BBU - https://phabricator.wikimedia.org/T184401#4292333 (10Reedy) [14:05:49] 10DBA: db1011 possibly faulty BBU - https://phabricator.wikimedia.org/T184401#4292561 (10Aklapper) p:05Lowest>03Normal [14:05:51] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1030 - https://phabricator.wikimedia.org/T184397#4292568 (10Aklapper) p:05Lowest>03Normal [14:09:28] 10DBA: db1011 possibly faulty BBU - https://phabricator.wikimedia.org/T184401#4292684 (10Aklapper) a:03Marostegui [14:09:36] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1030 - https://phabricator.wikimedia.org/T184397#4292685 (10Aklapper) a:03Cmjohnson [14:16:48] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T184285#4292801 (10Aklapper) [14:16:51] 10DBA, 10Patch-For-Review: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#4292803 (10Aklapper) [14:37:23] 10DBA, 10Cloud-Services, 10User-Urbanecm, 10cloud-services-team (Kanban): Prepare and check storage layer for inhwiki - https://phabricator.wikimedia.org/T184375#4293130 (10Aklapper) a:03Bstorm [15:11:32] 10DBA, 10Data-Services: Add base36 functions to ToolForge database - https://phabricator.wikimedia.org/T185673#4293324 (10jcrespo) 05Open>03stalled Could you please clarify my questions on my previous comment? Stalling until we get at response. [15:13:10] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#4293331 (10jcrespo) [15:13:13] 10DBA, 10Data-Services: Make Dispenser's principle_links table accessible in new Wiki replica cluster - https://phabricator.wikimedia.org/T180636#4293328 (10jcrespo) 05Open>03stalled Still waiting on getting the code to be able to deploy those to the replicas... [15:14:43] 10DBA, 10Phabricator, 10Release-Engineering-Team (Kanban), 10Security: Improve privilege separation for phabricator's config files and mysql credentials - https://phabricator.wikimedia.org/T146055#4293335 (10jcrespo) I am now more available, this should be added to the lists of things to discuss. [15:27:07] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293376 (10jcrespo) log_search is marked as a private table. We need 3 things here: * Security and/or the owner of that functionality to cla... [15:36:29] 10DBA, 10CheckUser, 10MediaWiki-Special-pages: Investigation: Add old and new length columns to cu_changes - https://phabricator.wikimedia.org/T155734#4293413 (10jcrespo) I don't think this is well thought, and I see some proposals here are not properly designed, at least from the point of view of WMF. Comme... [15:49:26] 10DBA, 10Wikidata, 10Performance, 10User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#4293458 (10jcrespo) I don't see huge issues on the current master- I would solve this as resolved and at some point thing about the epic parent (T108... [15:52:36] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4293484 (10jcrespo) a:03jcrespo [15:53:54] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4293487 (10Marostegui) Just FYI: I was waiting till after the offsite to merge the above patch and get rid of db1095. [16:00:31] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4293515 (10jcrespo) I wanted to prepare in advance the patches to clean up puppet (role deletions and all its dependencies). [16:01:31] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4293520 (10jcrespo) a:05jcrespo>03None Oh, I only saw the latest patch, nevermind. [16:07:19] 10DBA, 10CheckUser, 10MediaWiki-Special-pages: Investigation: Add old and new length columns to cu_changes - https://phabricator.wikimedia.org/T155734#4293534 (10MusikAnimal) 05Open>03declined This was originally requested as part of T145912, when we were intent on using `cu_changes` to add public IP ran... [16:11:45] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4293541 (10Marostegui) I think the patch that I still have pending to merge does it all, but I might have missed stuff! I would love to have a review :) [16:11:56] 10DBA, 10CheckUser, 10MediaWiki-Special-pages: Investigation: Add old and new length columns to cu_changes - https://phabricator.wikimedia.org/T155734#4293543 (10jcrespo) > As I'm sure you recall Don't be so sure :-), but kudos to me from the past, I guess. Thank you, I was just reviewing old requests, it i... [16:47:24] 10DBA, 10Wikidata, 10Performance, 10User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#4293640 (10Ladsgroup) That's another topic, I think we can call this resolved. [17:03:03] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#953798 (10Anomie) I don't know if anyone really "owns" the table, so I'll put my #mediawiki-platform-team hat on and look at it. Its purpose... [17:12:14] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293701 (10jcrespo) > I don't know if anyone really "owns" the table s/own/has any idea what the table is/ > If the table is exposed at all... [17:15:35] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293705 (10jcrespo) I just found an old comment by @Bawolff : > log_search: The information contained within would probably be useful to too... [17:39:36] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293742 (10Anomie) I can't see why there'd be password hashes in the table. `ls_value` will contain IPs for rows with `ls_field = 'target_au... [17:42:09] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293749 (10Bawolff) I agree with anomie's assesment. If ls_field is on a whitelist and we are sure that ls_log_id references something on the... [17:44:18] 10DBA, 10Data-Services, 10MediaWiki-General-or-Unknown, 10Security: Make (redacted) log_search table available on ToolLabs - https://phabricator.wikimedia.org/T85756#4293754 (10jcrespo) a:03jcrespo Thank you very much, I will now apply my changes and when I am done, move it to cloud. [17:57:53] 10DBA, 10Operations, 10Traffic, 10Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#4293777 (10jcrespo) [17:57:57] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#4293775 (10jcrespo) 05Open>03stalled This is stalled because tendril cannot work with multiple db backends. We would need to setup a different backend to support it- w... [18:01:01] 10DBA, 10MediaWiki-API, 10MediaWiki-Database: prop=revisions API timing out for a specific user and pages they edited - https://phabricator.wikimedia.org/T197486#4293785 (10Anomie) [18:08:44] 10DBA, 10MediaWiki-API, 10MediaWiki-Database: prop=revisions API timing out for a specific user and pages they edited - https://phabricator.wikimedia.org/T197486#4293684 (10jcrespo) I think what it is trying to do is to use one optimization for sorting when limit is used (priority queue). I will try to see i... [18:10:40] 10DBA, 10MediaWiki-API, 10MediaWiki-Database: prop=revisions API timing out for a specific user and pages they edited - https://phabricator.wikimedia.org/T197486#4293798 (10jcrespo) p:05Unbreak!>03High I don't think this is unbreak now, only a single combination of values have a performance issue- the fu... [18:50:51] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#4293838 (10Krinkle) [18:51:07] 10DBA, 10Operations, 10Traffic, 10Availability: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10Krinkle)