[00:48:53] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): SHOW EXPLAIN FOR via Quarry or sql-optimizer fails often - https://phabricator.wikimedia.org/T265430 (10AntiCompositeNumber) It looks like this has been a known issue for a while based on T195836/T205214, but the impact has been increased by backend cha... [05:09:08] 10DBA: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Ladsgroup) [05:13:17] 10DBA, 10Data-Persistence, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Marostegui) >>! In T261411#6539623, @Tgr wrote: > We are going to have separate tables for each wiki, r... [05:14:19] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10Marostegui) Excellent, thanks for the heads up. I haven't noticed anything on s5 (dewiki) or s6 (frwiki) - so we are good there [05:30:49] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): SHOW EXPLAIN FOR via Quarry or sql-optimizer fails often - https://phabricator.wikimedia.org/T265430 (10Marostegui) It makes some sense that it has been failing more often nowadays. We've been having load issues on the Analytics role, which ended up wit... [05:38:13] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) [05:38:28] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) a:03Marostegui [05:41:47] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) So this affects only `enwikivoyage.templatelinks` and `enwikivoyage.imagelinks`. The rest of wikis are ok for both tables. [05:41:57] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) p:05Triage→03Medium [06:14:35] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) Executed the following on `enwikivoyage.imagelinks`: ` alter table imagelinks drop key il_to, add key `il_to` (`il_to`,`il_from`); ` That tab... [06:15:38] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) [07:29:55] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) The following has been run on s5 master ` alter table templatelinks drop key tl_namespace, add key `tl_namespace` (`tl_namespace`,`tl_title`,`... [07:30:15] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [07:30:20] 10DBA, 10Data-Persistence: Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5 - https://phabricator.wikimedia.org/T265445 (10Marostegui) 05Open→03Resolved [07:31:45] 10DBA, 10Operations, 10ops-codfw, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) 05Open→03Resolved This host has been fully pooled and notifications were enabled too. Going to close this as fixed for now, we'll see if it crashes... [07:35:46] 10DBA, 10Patch-For-Review: Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts - https://phabricator.wikimedia.org/T165358 (10Marostegui) @Kormat can this be considered done as wmfmariadbpy scripts are being deployed via deb (ie: db-switchover,... [07:44:31] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) >>! In T265321#6537734, @Marostegui wrote: > So tinyint(4) happens on: > > s1, s2, some wikis of s5 (th... [07:50:05] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) [07:51:23] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) [07:54:50] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) [07:57:51] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) [08:18:02] 10DBA, 10Patch-For-Review: Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts - https://phabricator.wikimedia.org/T165358 (10Kormat) 05Open→03Resolved a:03Kormat @Marostegui: good point! Resolving. [08:28:47] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) [08:29:46] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Marostegui) @Ladsgroup s3 and s5 are fixed. The only pending thing is to push the patch to change this on code, so fu... [08:31:13] 10DBA, 10Data-Persistence, 10Schema-change, 10User-Ladsgroup: ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production - https://phabricator.wikimedia.org/T265321 (10Ladsgroup) Sure, I'm just waiting to finish the abstract schema change system merged so doing the patch would be seve... [08:32:52] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10Marostegui) Note: Depending on when the DB is created we might need to apply the alter from: {T265321} [08:37:53] sobanski: thanks for forwarding those wikitech-i mails to our list :) [13:05:30] marostegui: https://phabricator.wikimedia.org/P12993 - it's not the tidiest, but it's nice and clear now which section(s) a host is in [13:06:07] i can also make it cleaner as part of my work on multiinstance profiles [13:06:25] kormat: oh nice, that's very helpful indeed