[07:02:49] the *links schema change on commons is taking forever [07:06:32] big tables? [07:11:54] they are, but on innodb and dbstore it doesn't take so much [07:20:09] 10DBA, 10MediaWiki-Database, 06Performance-Team, 07Availability: wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged - https://phabricator.wikimedia.org/T95799#2631831 (10jcrespo) This is a more complex issue than it seems, aaron. Because we have "groups", if all... [07:32:29] 07Blocked-on-schema-change, 03Community-Tech-Sprint, 13Patch-For-Review, 07Schema-change, 05WMF-deploy-2016-09-13_(1.28.0-wmf.19): Add local_user_id and global_user_id fields to localuser table in centralauth database - https://phabricator.wikimedia.org/T141951#2631856 (10Marostegui) Hello, db1039 look... [07:43:43] 10DBA: Investigate (and if possible drop _counters) - https://phabricator.wikimedia.org/T145487#2631864 (10Marostegui) [08:12:40] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2631954 (10jcrespo) [08:13:47] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2627432 (10jcrespo) @Ladsgroup the title I added would have been more descriptive for me to understand the issue. I will be c... [08:20:09] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2631969 (10Ladsgroup) Thanks. [08:33:29] 10DBA, 10MediaWiki-Database, 06Performance-Team, 07Availability: wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged - https://phabricator.wikimedia.org/T95799#2632009 (10aaron) Yeah, there are complexities, which is why I never got around to it (though I looked a... [08:33:43] 10DBA, 10MediaWiki-Database, 06Performance-Team, 07Availability: wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged - https://phabricator.wikimedia.org/T95799#2632011 (10aaron) p:05High>03Normal [08:35:26] 10DBA: Investigate (and if possible drop _counters) - https://phabricator.wikimedia.org/T145487#2632012 (10Marostegui) The following has been executed in db1015: ``` for i in `mysql -hdb1015 -e "show databases;" -AB `; do echo ***$i***; mysql -hdb1015 $i -e "rename table _counters to TO_DROP__counters";done ```... [10:05:17] 10DBA, 06Operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#2632199 (10Marostegui) a:03Marostegui [10:33:34] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2632286 (10jcrespo) There are multiple rows with issue on production, here is the list I got some minutes ago: {P4035} [10:34:20] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2632289 (10jcrespo) a:05jcrespo>03None [10:36:46] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2632295 (10Ladsgroup) Okay, this is too big to be done manually. I think, I should write a maintenance script to do this job. [10:38:27] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2632300 (10jcrespo) I blocked myself to wait for your input, I am only standing by, but let me know when you need me again. [10:40:32] 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Check ORES data violating constraints on Beta do not affect production - https://phabricator.wikimedia.org/T145356#2632304 (10Ladsgroup) I will ask you soon. Thanks. [10:40:59] testwiki lives in S3, right? [10:45:35] yes [10:46:27] marostegui: I like using https://tools.wmflabs.org/replag/ to quickly figure out which wiki is on which shard [10:47:35] legoktm, marostegui https://noc.wikimedia.org/db.php [10:47:49] Thanks guys! And legoktm I was going thru db-eqiad.php but it wasn't obvious there, so next time I will use that one too [10:47:52] it is much compact [10:48:11] I remember jynus showing that to me before, but so much information so I forgot :) - bookmarked now [10:48:14] ah yeah, because it doesn't list all the s3 hosts :P [10:48:28] legoktm, with time you ended up knowing all shards by heart [10:48:34] haha [10:48:35] heh [10:48:52] Any reason not to show s3 ones? I guess because there are almost 1000? :) [10:49:06] because shard s3 does not exist [10:49:20] it is 'default' on mediawiki [10:49:44] Should I clarify this https://wikitech.wikimedia.org/wiki/MariaDB#Core_servers then? [10:49:56] no, that is mariadb [10:50:14] I was just answering the question of why it wasn't shown there [10:50:18] Aaaaah right right :) [10:50:53] we will probably soon move wikidata to s8 [10:51:00] and move other things around [10:53:00] Why? storage constraints? or just making it clearer? [10:54:58] availability concerns [10:57:46] ah ok :) [11:15:33] 10DBA, 10ChangeProp, 10MediaWiki-API, 10MediaWiki-Database, and 4 others: Investigate slow transcludedin query - https://phabricator.wikimedia.org/T145079#2632395 (10jcrespo) a:03jcrespo [12:20:00] 10DBA, 10MediaWiki-extensions-ContentTranslation, 05ContentTranslation-Release10, 05Language-Engineering July-September 2016, and 2 others: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2632576 (10Amire80) [13:51:18] 10DBA: Investigate (and if possible drop _counters) - https://phabricator.wikimedia.org/T145487#2632815 (10jcrespo) p:05Triage>03Normal [13:52:10] 07Blocked-on-schema-change, 10DBA: Deploy I2b042685 to all databases - https://phabricator.wikimedia.org/T139090#2632819 (10jcrespo) p:05Triage>03Normal a:03jcrespo [13:54:03] 10DBA, 06Operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#2632821 (10Marostegui) enwiki -> s1 testwiki ->s3 commonswiki -> s4 non pooled slaves to do some testing before dropping the table for good s1 -> db1073 s3 -> db1044 s4 -> db10... [16:14:54] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2633433 (10Marostegui) [16:29:51] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2633472 (10Marostegui) @jcrespo saw a kernel panic when he logged via console. At a quick glance: Also the server has been showing kernel errors lately (for the last few days: ``` Sep 13 11:06:17 db1082 kernel: [888031... [16:42:05] 10DBA, 10CatWatch, 10MediaWiki-General-or-Unknown, 06TCB-Team, 07Wikimedia-log-errors: SELECT /* CategoryMembershipChangeJob::run 127.0.0.1 */ GET_LOCK('CategoryMembershipUpdates:XXXX', 10) AS lockstatus - https://phabricator.wikimedia.org/T133801#2633503 (10Addshore) [16:43:48] 10DBA, 10CatWatch, 10MediaWiki-General-or-Unknown, 06TCB-Team, 07Wikimedia-log-errors: SELECT /* CategoryMembershipChangeJob::run 127.0.0.1 */ GET_LOCK('CategoryMembershipUpdates:XXXX', 10) AS lockstatus - https://phabricator.wikimedia.org/T133801#2244629 (10Addshore) Would it make sense to not log these... [17:14:08] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2633623 (10jcrespo) I am adding Moritz, not expecting to do anything here, but just a heads up incase he is aware of any recent kernel issue and we are behind in updates for this server. Let's do proper debugging tomorrow. [17:20:43] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2633663 (10dduvall) We had to abort the migration due to time constr... [17:27:28] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2633717 (10jcrespo) @dduvall I leave you here the full tutorial for... [17:30:35] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2633733 (10dduvall) >>! In T138778#2633717, @jcrespo wrote: > @dduva... [17:35:26] 10DBA, 10ChangeProp, 10MediaWiki-API, 10MediaWiki-Database, and 4 others: Investigate slow transcludedin query - https://phabricator.wikimedia.org/T145079#2633741 (10jcrespo) I regenerated the table statistics without sucess, I will try to tune the query planner next forcing the usage of histograms. ``` Ma... [18:58:55] 10DBA, 10CatWatch, 10MediaWiki-General-or-Unknown, 06TCB-Team, 07Wikimedia-log-errors: SELECT /* CategoryMembershipChangeJob::run 127.0.0.1 */ GET_LOCK('CategoryMembershipUpdates:XXXX', 10) AS lockstatus - https://phabricator.wikimedia.org/T133801#2634248 (10hashar) Our Puppet manifest for HHMV has: `sl... [22:42:55] 10DBA, 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 07WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2635181 (10AndyRussG) [23:27:45] 07Blocked-on-schema-change, 06Community-Tech, 13Patch-For-Review, 07Schema-change, 05WMF-deploy-2016-09-13_(1.28.0-wmf.19): Add local_user_id and global_user_id fields to localuser table in centralauth database - https://phabricator.wikimedia.org/T141951#2635347 (10DannyH)