[01:22:48] DBA, Operations, Phabricator: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2416754 (mmodell) @jcrespo: I say go ahead with the m3 slave depooling. AFAIK the slaves are only used (currently) for some expensive analytics-type queries (the public task dump) This will... [01:33:16] DBA, Operations, Phabricator: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2416755 (mmodell) @JAufrecht: adding you just to give you a bit of fore-warning that the public task dump is going to break sometime this week, not sure how long before we can restore it but... [01:54:22] DBA, ArchCom-RfC, Schema-change: [RFC] Use index on rc_this_oldid - https://phabricator.wikimedia.org/T139012#2416768 (Ladsgroup) [01:55:11] DBA, ArchCom-RfC, Schema-change: [RFC] Use index on rc_this_oldid - https://phabricator.wikimedia.org/T139012#2416783 (Ladsgroup) [10:48:18] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2417391 (hashar) I have myself have no idea how to enable strict mode / which part of puppet c... [11:03:03] DBA, Beta-Cluster-Infrastructure, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2417424 (hashar) Might MySQL not sending any log at all. On deployment-db1 it runs as: ``` /usr/sbin/mysqld \ --basedir=/usr -... [11:07:42] DBA, Patch-For-Review: Improve db backup handling, specially of misc hosts - https://phabricator.wikimedia.org/T138562#2417447 (jcrespo) [11:07:44] DBA, Patch-For-Review: dbstore1001 low available space - https://phabricator.wikimedia.org/T131705#2417444 (jcrespo) Open>Resolved a:jcrespo ``` /dev/mapper/tank-data 6.2T 5.2T 1.1T 84% /srv ``` [11:11:53] DBA, Beta-Cluster-Infrastructure, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2417450 (hashar) The db1 instance has puppet class `role::mariadb::beta` which invokes: ``` lang=ruby class { 'mariadb::confi... [11:20:56] DBA, Beta-Cluster-Infrastructure, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2417464 (jcrespo) > ./templates/mariadb/beta.my.cnf.erb yes, if it is not in use by any other server, just add it there. [11:22:42] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL strict mode on CI slaves - https://phabricator.wikimedia.org/T119371#2417466 (jcrespo) The technical parts of these are just adding the config to `beta.... [11:23:48] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2417467 (jcrespo) This is not blocked by that- it only needs to be enabled on config (see my c... [11:28:21] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2417470 (jcrespo) This affects to more things than just that cron, I had to revert: https://gerrit.wikimedia.org/r/296709 . I will create an alternative proposal to depo... [11:33:49] DBA, Beta-Cluster-Infrastructure, Patch-For-Review, WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#2417485 (hashar) a:hashar [13:39:27] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2409864 (hashar) [13:42:26] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2417830 (hashar) I have repurposed this task to solely track the upgraded of #beta-cluster databases to Jessie/Mar... [13:43:54] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2417831 (hashar) [13:45:00] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2409864 (hashar) [13:45:02] DBA, Performance-Team, Availability, Epic, Patch-For-Review: MASTER_POS_WAIT() alternative that works cross-DC - https://phabricator.wikimedia.org/T135027#2417835 (hashar) [13:55:55] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2417860 (hashar) After some discussion with @jcrespo he will be able to do the migration... [14:00:03] DBA, Beta-Cluster-Infrastructure, Release-Engineering-Team, WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2417888 (hashar) [14:34:01] DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2418098 (jcrespo) [14:34:24] DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2418110 (jcrespo) a:jcrespo Testing on db1073. [14:35:38] DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2418113 (jcrespo) [14:37:40] DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2418115 (jcrespo) db1073, at ~14 UTC, 2016-06-30, before compression: ``` +--------------------+----------------------------------------------------+--------------------+----------+---------+---------+---------+---------+ | DATABASE... [14:38:47] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418117 (Umherirrender) [14:44:17] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418117 (jcrespo) @Umherirrender when was the last time (aproximately) that this call gave you an error? I cannot find it on the logs, but I may just need to ch... [14:44:47] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418117 (Anomie) Looks like yet another database timeout error. [14:47:36] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418184 (Anomie) ``` 2016-06-30 14:31:30 [V3UtBgpAANIAAByo3I4AAACV] mw1207 dewiki 1.28.0-wmf.7 exception ERROR: [V3UtBgpAANIAAByo3I4AAACV] /w/api.php?action=que... [14:52:02] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418192 (Umherirrender) >>! In T139056#2418177, @jcrespo wrote: > @Umherirrender when was the last time (aproximately) that this call gave you an error? I canno... [15:12:24] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418303 (jcrespo) Ah, I could not find it because this is not a "db error" but an exception, of which there is not much difference, but it goes to a different l... [15:16:50] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418312 (Anomie) I note that an explain for that query seems to be choosing the wrong index: ``` +------+-------------+-----------+--------+--------------------... [15:19:45] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418362 (jcrespo) @Umherirrender May I ask you to test with a lower lhlimit ? I suspect there could be a limit in which it hits a tipping point where the databa... [15:22:23] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418370 (jcrespo) > it's for some reason choosing the pl_backlinks_namespace index over pl_namespace. That is easily explainable by a very large IN- large INs... [15:24:47] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418377 (Anomie) When I execute the query manually, it took 1 minute 3.31 seconds to return the results. Executing it with `USE INDEX (pl_namespace)` took a tin... [15:43:35] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418463 (jcrespo) The issue is: https://www.percona.com/blog/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/ It was fixed on MySQL 5.6: https:/... [15:52:42] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418515 (Umherirrender) >>! In T139056#2418362, @jcrespo wrote: > @Umherirrender May I ask you to test with a lower lhlimit ? I suspect there could be a limit i... [15:59:58] DBA, MediaWiki-API: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418542 (jcrespo) @Umherirrender We are not going to ask you to do that, this is clearly an optimizer issue (although feel free to do that while this is solved)... [16:05:17] jynus: Possibly stupid question. I know we're in progress on m3 migration....has m2 been done yet I can't remember? [16:06:38] no, m2 migration is much less priority [16:06:55] because it is on trusty and MariaDB 10 already [16:07:03] it would be nice to do some upgrades [16:07:12] but way less critical [16:07:30] and needs more preparation because there there several things aside from reviewdb there [16:08:07] m1 and m3 where the priority ones because they were the last ones on precise/5.5 [16:09:44] trusty/10.0.15 -> jessie/10.0.23 has less compatibility issues and it is not a short-term goal (it will be done slowly) [16:26:53] DBA, MediaWiki-API, Upstream: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2418613 (jcrespo) https://jira.mariadb.org/browse/MDEV-10313 [16:31:43] jynus: Makes sense, thanks [17:03:10] DBA, Performance-Team, Availability, MW-1.27-release-notes, and 2 others: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) - https://phabricator.wikimedia.org/T111266#2418782 (jcrespo) I think I have an alternative solution, without requiring exte... [17:42:44] DBA, MediaWiki-API, Upstream: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2419026 (Anomie) @jcrespo: I wonder if this would go away if https://gerrit.wikimedia.org/r/#/c/190774/ had actually been deployed to (all) the da... [17:48:01] DBA, MediaWiki-API, Upstream: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2419033 (jcrespo) Could you please file a ticket to check that (a #blocked-on-schema-change)? There may be hundreds of pending changes that were n... [17:58:42] Blocked-on-schema-change: Deploy I2b042685 to all databases - https://phabricator.wikimedia.org/T139090#2419082 (Anomie) [17:59:13] DBA, MediaWiki-API, Patch-For-Review, Upstream: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2419112 (Anomie) >>! In T139056#2419033, @jcrespo wrote: > Could you please file a ticket to check that (a #blocked-on-schem... [18:03:11] Blocked-on-schema-change, DBA: Deploy I2b042685 to all databases - https://phabricator.wikimedia.org/T139090#2419123 (jcrespo) [18:03:46] DBA, MediaWiki-API, Patch-For-Review, Upstream: API database error on continue with generator=categorymembers and prop=linkshere - https://phabricator.wikimedia.org/T139056#2419125 (jcrespo) Thank you. [20:30:03] DBA: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2419687 (ArielGlenn) This is a Big Deal. Or rather, I should say that T132416 and its ilk are a Big Deal. It's not just enwiki of course; it's all of them....