[07:09:19] DBA, Beta-Cluster-Infrastructure, Flow, Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2384612 (jcrespo) Go for it. You may want to have some kind of check/flowdb list usage maybe, thinking in production? I think... [08:46:58] DBA: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#2384719 (jcrespo) [08:47:00] DBA, Operations: High replication lag to dewiki - https://phabricator.wikimedia.org/T135100#2384720 (jcrespo) [08:47:02] DBA, Operations, Patch-For-Review: reimage or decom db servers on precise - https://phabricator.wikimedia.org/T125028#2384721 (jcrespo) [08:47:05] DBA, MediaWiki-Database: Apply recentchanges special partitioning to a redundant slave at least on s1, s4, and s5 - https://phabricator.wikimedia.org/T120513#2384722 (jcrespo) [08:47:08] DBA, Operations: Physical location SPOF because of database server distribution on a single rack (D1) - https://phabricator.wikimedia.org/T111992#2384723 (jcrespo) [08:47:11] DBA, Operations, Patch-For-Review: Install, configure and provision recently arrived db core machines - https://phabricator.wikimedia.org/T133398#2384717 (jcrespo) Open>Resolved All 16 new servers (21 in total, 3 per shard) are pooled into production- we will do some adjustments over the foll... [08:50:48] DBA, Operations, Epic: Eliminate SPOF at the main database infrastructure - https://phabricator.wikimedia.org/T119626#2384731 (jcrespo) [08:50:50] DBA, Operations: Physical location SPOF because of database server distribution on a single rack (D1) - https://phabricator.wikimedia.org/T111992#2384728 (jcrespo) Open>Resolved a:jcrespo This is now fixed, D1 is no longer a SPOF. Although somehow heavy, if D1 or the whole D row went down, we... [08:52:31] DBA: Heartbeat check for old 5.5 master is not checking the lag - https://phabricator.wikimedia.org/T133329#2384734 (jcrespo) Open>declined All old servers (except db1053, which was decommed) have now been reinstalled. [08:58:14] DBA, Labs, Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#2384740 (jcrespo) [08:58:49] DBA, Labs, Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#2198300 (jcrespo) labsdb1003 is now also at 10% free space. [11:37:34] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2384976 (jcrespo) I have now the watchlist count generator running on all public wikis. I will measure how much it takes for a second run (after indexes are hot) on all wikis to establish th... [11:37:52] DBA, Labs, Labs-Infrastructure, Operations, Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2286195 (jcrespo) [11:37:56] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2384978 (jcrespo) [11:38:08] Blocked-on-schema-change, DBA, Expiring-Watchlist-Items, MediaWiki-Watchlist, TCB-Team: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2384981 (jcrespo) [11:38:10] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#609458 (jcrespo) [11:42:13] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2382984 (Ladsgroup) a:Ladsgroup [12:45:00] DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2385068 (jcrespo) [12:52:51] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385087 (Ladsgroup) By using INNER JOIN instead of LEFT JOIN in case of hidenondamaging, everything would tons and tons of times faster. [13:37:21] DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2385207 (jcrespo) I think we can get acceptable performance numbers if we sell "our souls" a little- aka introducing some complexity that will requir... [13:50:04] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385248 (jcrespo) Can you copy me the potential SQL queries generated by your code so I can run EXPLAIN on them on production to test them? Thank you. [13:56:07] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review: hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385256 (Ladsgroup) ``` EXPLAIN SELECT /* SpecialRecentChanges::doMainQuery Ladsgroup */ rc_id,rc_timestamp,rc_user,rc_user_text,rc_namespace,rc_ti... [14:02:32] DBA, Datasets-General-or-Unknown: Select of revisions for stub history files does not explicitly order revisions - https://phabricator.wikimedia.org/T29112#2385265 (ArielGlenn) Thanks a lot for this. Because the number of revisions per page varies, with e.g. the first 30k pages in en wikipedia having 24... [14:14:36] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review, WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385293 (jcrespo) I think the changes look good, 30K-100K rows visited using an index. It could be better,... [14:30:20] DBA, Wikidata, Patch-For-Review, Wikidata-Sprint-2016-05-24: Wikidata master database connection issue - https://phabricator.wikimedia.org/T136598#2385377 (jcrespo) Open>Resolved Master connection issues not seen for a week. [14:38:31] DBA, Labs, Tool-Labs: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2228235 (Stevietheman) What are the current development needs of this tool? I'm really missing its output in projects I'm associated with. (note that I'm not com... [14:40:51] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review, WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385421 (Ladsgroup) @jcrespo I wrote there, I hope I understood you correctly, We can't get that threshold... [14:48:15] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review, WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385445 (jcrespo) No, I am talking about using a particular role when connecting to the database and selec... [14:51:44] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review, WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385459 (Ladsgroup) Oh, this hook is not changing the DB slave role, It only adds condition and stuff, the... [15:40:17] DBA, MediaWiki-extensions-ORES, Revision-Scoring-As-A-Service, Patch-For-Review, WMF-deploy-2016-06-21_(1.28.0-wmf.7): hidenondamaging is slow - https://phabricator.wikimedia.org/T137895#2385610 (jcrespo) Open>Resolved [15:57:48] DBA, Labs, Tool-Labs: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2385640 (kaldari) > did you get a chance at further deletion @kaldari? @chasemp: I've deleted everything older than 2011 now. Can you see how much disk space is b... [16:00:45] DBA, Labs, Tool-Labs: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2385658 (jcrespo) > Can you see how much disk space is being used currently? ``` du -hs p50380g50816__pop_stats 50G p50380g50816__pop_stats ``` [20:42:18] DBA, Beta-Cluster-Infrastructure, Flow, Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2386806 (Mattflaschen-WMF) >>! In T119567#2384612, @jcrespo wrote: > Go for it. You may want to have some kind of check/flowdb... [22:04:48] DBA, Labs, Operations, Tool-Labs, Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2376097 (Legoktm) >>! In T137707#2382924, @Antigng_ wrote: > Lack of hard and fast limit on read requests can be a problem, since your definition of request limit... [22:10:55] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2387246 (PleaseStand) [22:12:27] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1516355 (PleaseStand) [22:12:43] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1516355 (PleaseStand) [22:12:54] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1516355 (PleaseStand) [22:13:53] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2387259 (PleaseStand) [22:20:44] DBA, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure, MediaWiki-Database, WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2387273 (RobLa-WMF) This is currently unassigned. Is that on purpose? I'm not sure if @hasha... [23:08:49] DBA, Collaboration-Team-Interested, Flow, Operations, WorkType-Maintenance: Setup separate logical External Store for Flow - https://phabricator.wikimedia.org/T107610#2387618 (Mattflaschen-WMF) a:jcrespo @jcrespo, I think this is the next concrete step ({T119568} will get QA-ed, but that'... [23:17:01] DBA, Collaboration-Team-Interested, Flow, Operations, WorkType-Maintenance: Setup separate logical External Store for Flow - https://phabricator.wikimedia.org/T107610#2387654 (Mattflaschen-WMF) [23:18:47] DBA, Collaboration-Team-Interested, Flow, Operations, WorkType-Maintenance: Setup separate logical External Store for Flow in production - https://phabricator.wikimedia.org/T107610#1499219 (Mattflaschen-WMF)