[04:07:31] 10DBA, 10MediaWiki-extensions-Linter: Purge html5-misnesting tags from the database - https://phabricator.wikimedia.org/T178040#3678602 (10ssastry) [04:07:39] 10DBA, 10MediaWiki-extensions-Linter: Purge html5-misnesting tags from the database - https://phabricator.wikimedia.org/T178040#3678617 (10ssastry) p:05Triage>03High [04:08:52] 10DBA, 10MediaWiki-extensions-Linter: Purge html5-misnesting tags from the database - https://phabricator.wikimedia.org/T178040#3678602 (10ssastry) If @Legoktm is busy, would the dba folks be able to help with this? [06:12:04] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3678688 (10awight) >>! In T171027#3677449, @Catrope wrote: >... [07:20:24] 10DBA, 10MediaWiki-extensions-Linter: Purge html5-misnesting tags from the database - https://phabricator.wikimedia.org/T178040#3678744 (10Legoktm) The script is running in a screen on terbium. The query is `DELETE FROM linter WHERE linter_cat=12 ORDER BY linter_id LIMIT 500` running in a loop until the query... [10:26:58] 10DBA, 10MediaWiki-Platform-Team, 10monitoring, 10Performance-Team (Radar): Improve database application performance monitoring visibility - https://phabricator.wikimedia.org/T177778#3679076 (10Aklapper) [11:06:34] 10DBA, 10MediaWiki-Watchlist, 10Wikidata: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) - https://phabricator.wikimedia.org/T177772#3679132 (10jcrespo) ``` root@db2076[ruwiki]> SELECT COUNT(*) FROM recentchanges WHERE rc_source = 'wb... [11:14:25] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3679139 (10jcrespo) This is the result on query error rate af... [11:15:30] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Russian-Sites: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) - https://phabricator.wikimedia.org/T177772#3679142 (10Liuxinyu970226) [11:41:24] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3679191 (10jcrespo) Backups of local hosts worked, but x1 only backed up %wik% databases, we have to get a full list of databases there and add it to dump... [12:09:54] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3679310 (10eranroz) 1, @hoo so how do we move forward with deployment to larger wikis/wikis with more wikidata usage... [13:41:25] 10DBA, 10monitoring, 10Epic, 10Wikimedia-Incident: Reduce false positives on database pages - https://phabricator.wikimedia.org/T177782#3679652 (10jcrespo) > Do you have suggestion(s) on what functionality (or URL) in MediaWiki to do a monitoring check against, which would uncover most or all DB issues whi... [14:18:02] 10DBA: Unwriteable error when creating DR in Commons:Data - https://phabricator.wikimedia.org/T178051#3679748 (10Bawolff) I wonder if its related to deleting all the wikibase stuff in rc(?) [14:42:46] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3679801 (10hoo) >>! In T151717#3679310, @eranroz wrote: > 1, @hoo so how do we move forward with deployment to larger... [16:47:10] 10DBA, 10Operations, 10ops-eqiad: Decommission db1024 - https://phabricator.wikimedia.org/T164702#3680249 (10Cmjohnson) [16:48:28] 10DBA, 10Operations, 10Patch-For-Review: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) - https://phabricator.wikimedia.org/T162699#3680252 (10Cmjohnson) [16:48:31] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3680253 (10Cmjohnson) [16:48:33] 10DBA, 10Operations, 10ops-eqiad: Decommission db1024 - https://phabricator.wikimedia.org/T164702#3242621 (10Cmjohnson) 05Open>03Resolved done [16:49:22] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3680271 (10Cmjohnson) [17:06:35] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3680285 (10Cmjohnson) [17:06:37] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3680284 (10Cmjohnson) 05Open>03Resolved [17:28:19] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680450 (10Bawolff) [17:28:35] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680466 (10Bawolff) [17:29:19] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680450 (10Bawolff) [17:31:35] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Russian-Sites: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) - https://phabricator.wikimedia.org/T177772#3680509 (10Bawolff) Hi, There's been reports of high amounts of lag on commons cau... [17:44:12] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680450 (10Legoktm) This could have been caused by {T178040}, which disproportionately affected commonswiki? I started it at 7:18 UTC according to SAL and commons was the first large wiki it hit... [17:44:33] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3680537 (10Catrope) >>! In T171027#3677449, @Catrope wrote: >... [18:03:40] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3680596 (10Johan) @jcrespo Is this an accurate description ri... [18:09:17] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3680606 (10IKhitron) Hi, @Johan. Recent changes or watchlists? [18:11:12] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3680612 (10Catrope) >>! In T171027#3680606, @IKhitron wrote:... [18:12:09] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3680613 (10Johan) Will fix. [18:17:58] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680643 (10Bawolff) [18:22:04] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680648 (10Bawolff) [19:40:01] 10DBA, 10Commons: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3680450 (10Marostegui) The rc purge was stopped today at 9:18 UTC so it is unlikely it caused it. I would be more inclined to think that it was due to @Legoktm script. As why it didn't alert, w... [19:40:48] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Russian-Sites: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) - https://phabricator.wikimedia.org/T177772#3680916 (10Marostegui) I believe this is more likely to be the cause: T178094#3680533 [20:05:48] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3680984 (10bd808) a:03Andrew This should be ready for the https://wikitech.wikimedia.org/wiki/Add_a_wiki#Cloud_Services steps. [20:14:22] 10DBA, 10Data-Services, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3681008 (10bd808) [20:14:52] 10DBA, 10Data-Services: Provide a new sanitarium host for s8 - https://phabricator.wikimedia.org/T177274#3681009 (10bd808) [20:20:28] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Identify tools hosting databases on labsdb100[13] and notify maintainers - https://phabricator.wikimedia.org/T175096#3681046 (10bd808) [20:23:21] 10DBA, 10Community-Tech, 10Data-Services, 10Security: Create core ip_changes view for replicas - https://phabricator.wikimedia.org/T173891#3681051 (10bd808) >>! In T173891#3546336, @Bawolff wrote: >> Should the data be made available on the labs replicas and/or dumps: Yes, nothing in the table is private d... [20:36:25] 10DBA, 10Operations: Wikimedia\Rdbms\DBQueryTimeoutError (not repeated) - https://phabricator.wikimedia.org/T178109#3681065 (10Base) [20:40:56] 10DBA, 10Data-Services, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3681083 (10MusikAnimal) This seems to have done the trick. The last max connections error on the API server was on 2017-10-09, as reporte... [22:47:49] 10DBA, 10Data-Services, 10Security-Team: pagetranslation log_type missing on replicas - https://phabricator.wikimedia.org/T178052#3681564 (10bd808) The [[https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/templates/labs/db/views/maintain-views.yaml;1cc8cd81639bbddf23f0a... [23:17:47] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team: Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3681627 (10bd808)