[00:52:42] 10DBA, 10MediaWiki-ResourceLoader, 07Wikimedia-log-errors: Deadlock found when trying to get lock; try restarting transaction - https://phabricator.wikimedia.org/T117089#2497599 (10Krinkle) [00:53:09] 10DBA, 10MediaWiki-ResourceLoader, 07Wikimedia-log-errors: Deadlock found when trying to get lock; try restarting transaction - https://phabricator.wikimedia.org/T117089#1766301 (10Krinkle) [00:53:44] 10DBA, 10MediaWiki-ResourceLoader, 07Wikimedia-log-errors: Deadlock found when trying to get lock; try restarting transaction - https://phabricator.wikimedia.org/T117089#1766301 (10Krinkle) [00:53:47] 10DBA, 10MediaWiki-General-or-Unknown, 05MW-1.27-release-notes, 13Patch-For-Review, 07Wikimedia-log-errors: "Lock wait timeout exceeded" errors regarding module_deps - https://phabricator.wikimedia.org/T124649#2497605 (10Krinkle) [00:55:29] 10DBA, 10MediaWiki-ResourceLoader, 07Wikimedia-log-errors: Deadlock found when trying to get lock; try restarting transaction - https://phabricator.wikimedia.org/T117089#2497606 (10Krinkle) There are no results for `module_deps` in mediawiki-errors and wfLogDBError for the last 7 days, except about a dozen h... [01:34:46] 10DBA, 10MediaWiki-ResourceLoader, 07Wikimedia-log-errors: Deadlock found when trying to get lock; try restarting transaction - https://phabricator.wikimedia.org/T117089#2497664 (10Krinkle) 05Open>03Resolved a:03Krinkle Closing this for now. The aforementioned patches should address this once they roll... [02:37:58] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2497829 (10Dzahn) [02:39:04] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2491652 (10Dzahn) ``` [terbium:~] $ which sql /usr/local/bin/sql [terbium:~] $ cat /usr/local/bin/sql #!/bin/bash # This file is managed by Puppet (modules/scap/files/sql). ``` [02:40:20] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2497832 (10Dzahn) ``` # Look up MySQL host to connect to. For centralauth the host cannot # be determined this way, so we need to use fawiki instead as it is # located on the same server in both pro... [03:08:33] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2497839 (10Dzahn) @Amire80 on terbium, in your home dir in .bashrc , add this code: https://gerrit.wikimedia.org/r/#/c/301326/1/modules/admin/files/home/amire80/.bashrc then "source .bashrc" and th... [03:09:46] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2497852 (10Dzahn) [terbium:~] $ vi .bashrc [terbium:~] $ source .bashrc dzahn@terbium:~$ sqlhost dewiki db1092 [03:10:39] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2497854 (10Dzahn) p:05Triage>03Normal [06:10:34] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2498003 (10Amire80) Thanks, but the point is that I already made such tricks for my own account, and it would be useful to have the same for other accounts. I have several scripts that I run there re... [06:35:19] 10DBA, 10Analytics-EventLogging, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2497306 (10jcrespo) This means dropping: ``` ImageMetricsLoadingTime_10078363 ImageMetricsCorsSupport_10884476 ImageMetricsCorsSupport... [06:40:20] 10DBA, 10Analytics-EventLogging, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2497306 (10jcrespo) [06:40:23] 10DBA, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2498021 (10jcrespo) [06:51:03] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2491652 (10jcrespo) > if I need to connect to a host from another script in a way that doesn't involve running the mysql command as such. > I have several scripts that I run there regularly Could y... [06:57:17] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2498032 (10Amire80) >>! In T141255#2498028, @jcrespo wrote: >> if I need to connect to a host from another script in a way that doesn't involve running the mysql command as such. > >> I have several... [07:09:42] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2498051 (10jcrespo) > https://gerrit.wikimedia.org/r/#/c/282312/3/bash/published_despite_errors.py That looks like something you shouldn't run from terbium (but more analytics-research-y), but from... [07:14:20] 10DBA, 06Operations: Separate host lookup from the sql shell script - https://phabricator.wikimedia.org/T141255#2498063 (10jcrespo) Answer your original question, these are the easy dns entries that you should probably be using, which makes things easier, and generally points to the right hosts on each occasio... [07:29:40] 10DBA, 10Analytics-EventLogging, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2498084 (10jcrespo) p:05Triage>03Normal [08:51:31] 10DBA, 10Dumps-Generation, 13Patch-For-Review: db1053 transient replica lag during dumping - https://phabricator.wikimedia.org/T132279#2498209 (10ArielGlenn) This will kick off on the 1st or 2nd of August and I'll be watching it. [09:39:48] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2498272 (10jcrespo) > Switching back to Open per above, feel free to apply the schema change to all wikis if you prefer so. You haven't discussed t... [10:48:57] 10DBA, 06Operations, 13Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2498391 (10jcrespo) All clients are using now dbproxy1006; except bacula. I will wait for backup jobs to finish to restart it before reimaging dbproxy1001. [14:19:03] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2498845 (10jcrespo) I can confirm this is a very rare occurence, only 50 times in a week. Explain on db1057: ``` *********************... [14:29:32] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2498866 (10jcrespo) Forcing `log_title_type_time` on logging seems to create the same query plan, and a much faster query. The reason fo... [14:50:13] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2498923 (10jcrespo) However, if I `IGNORE INDEX(type_time)`, it goes with `page_time`, and still ignores `log_title_type_time`. I have t... [15:12:35] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2498992 (10Nemo_bis) Are you saying there is a policy of "schema changes are applied only when no alternative is conceivably possibly to code"? This... [15:23:48] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2499035 (10jcrespo) The policy is: schema changes on production are not applied unless there is general consensus among mediawiki developers about t... [16:04:27] 10DBA, 10Analytics, 13Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2499147 (10Nuria) [17:32:18] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2499479 (10jcrespo) The server is running an older version of MariaDB, unlike the others, that could be related, too. I will also upgrad... [18:00:18] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2499603 (10Amire80) This continues, with 74–93 errors each day since I created the bug. There were no... [18:16:18] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2486720 (10jcrespo) Assuming this is x1, I cannot see any time those server lagged more than 2 seconds... [20:27:14] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2500117 (10Nemo_bis) > The policy is: schema changes on production are not applied unless there is general consensus among mediawiki developers abou... [20:51:18] 10DBA, 10Analytics, 13Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2500140 (10Neil_P._Quinn_WMF) @Nuria, @mforns, I have updated the documentation at [wikitech:Analytics/EventLogging](https://wikitech.wikimedia.org/wiki/Analytics/EventLogging#Data_ret... [22:39:16] 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2500509 (10greg) [22:40:20] 10DBA, 13Patch-For-Review, 05Wikimedia-Incident: Improve db backup handling, specially of misc hosts - https://phabricator.wikimedia.org/T138562#2500515 (10greg)