[06:01:55] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3274600 (10Marostegui) labsdb1001 is done: ``` [root@labsdb1001 05:57 /home/marostegui # for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl $i -e "show create table revision\G";done bgwiki ***... [06:30:29] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3274615 (10Marostegui) `enwiki` db1051 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb1051 enwik... [06:31:09] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3274616 (10Marostegui) db1051 has been optimized: ``` root@db1051:/srv/sqldata/enwiki# ls -lh ores_class... [07:32:48] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3274666 (10Marostegui) db1067 has been optimized: ``` root@db1067:/srv/sqldata/enwiki# ls -lh ores_class... [07:32:50] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3274667 (10Marostegui) `enwiki` db1067 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb1067 enwik... [07:38:36] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3274688 (10Marostegui) In order to checksum the only pending database of s7 (`frwiktionary`) we need to either ignore the revision table (see: T163190#3250637) or fix it. I will for now, create the task to unify the revision table on... [09:54:17] 10DBA, 10MediaWiki-Database, 07Technical-Debt: Merge pagelinks, templatelinks and imagelinks tables - https://phabricator.wikimedia.org/T161066#3275206 (10EddieGP) Before we go further into this I think some feedback with "Yes, this would be senseful, go ahead and discuss" or "No, the new table would be much... [10:16:06] 10DBA, 10MediaWiki-Database, 07Technical-Debt: Merge pagelinks, templatelinks and imagelinks tables - https://phabricator.wikimedia.org/T161066#3120487 (10jcrespo) You added DBA, which means you want our perspective from the point of view of being in charge of a site with high scalability and performance req... [12:21:48] 10DBA: frwiktionary on s7 still needs fixing on the revision table - https://phabricator.wikimedia.org/T165743#3275703 (10Marostegui) [12:22:12] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3275722 (10Marostegui) [12:22:14] 10DBA: frwiktionary on s7 still needs fixing on the revision table - https://phabricator.wikimedia.org/T165743#3275721 (10Marostegui) [12:22:41] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3189184 (10Marostegui) [12:26:02] 10DBA: frwiktionary on s7 still needs fixing on the revision table - https://phabricator.wikimedia.org/T165743#3275741 (10Marostegui) [12:44:34] 10DBA, 10MediaWiki-Database, 07Technical-Debt: Merge pagelinks, templatelinks and imagelinks tables - https://phabricator.wikimedia.org/T161066#3120487 (10Marostegui) Just to say that I agree 100% with Jaime on this. Having another mega table (we already have revision and some others) would make our (DBA) li... [13:36:53] 10DBA, 10Wikimedia-Hackathon-2017, 10Wikimedia-Site-requests, 07Documentation, 05Mediawiki SWAT Deployments: Create summary templates to stop to write the same things everywhere everytime - https://phabricator.wikimedia.org/T165756#3276060 (10Dereckson) [13:37:09] 10DBA, 10Wikimedia-Hackathon-2017, 10Wikimedia-Site-requests, 07Documentation, 05Mediawiki SWAT Deployments: Create summary templates to stop to write the same things everywhere everytime - https://phabricator.wikimedia.org/T165756#3276060 (10Dereckson) a:05Dereckson>03None [13:37:57] 10DBA, 10Wikimedia-Hackathon-2017, 10Wikimedia-Site-requests, 07Documentation, 05Mediawiki SWAT Deployments: Create summary templates to stop to write the same things everywhere everytime - https://phabricator.wikimedia.org/T165756#3276060 (10Dereckson) [14:25:01] 10DBA, 13Patch-For-Review: frwiktionary on s7 still needs fixing on the revision table - https://phabricator.wikimedia.org/T165743#3276353 (10Marostegui) db2068 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb2068.codfw.wmnet frwiktionary -e "show create table revision\... [14:25:41] 2037? [14:25:54] and 51 [14:26:00] looks like there was a spike of lag in s4 [14:26:34] but codfw only? [14:27:04] so far yes [14:27:06] I am checking codfw master [14:27:29] global: https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?panelId=4&fullscreen&orgId=1&from=now-1h&to=now [14:27:48] Ah, I was checking right now [14:28:00] So I guess whatever it was is being replciated to codfw now :( [14:28:34] makes sense for a second later to be slowe [14:28:35] r [14:28:39] *layer [14:29:25] the master is no longer delayed [14:29:28] (codfw master) [14:30:19] inserts and updates multiplied by 100 [14:30:49] it is not infra, it is app [14:31:04] https://grafana.wikimedia.org/dashboard/db/mysql?panelId=2&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1068 [14:31:20] :O [14:32:36] I am checking SAL and there is nothing there that suggests something was enabled at that time [14:50:43] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276493 (10jcrespo) 05declined>03Open This just happened again on s4. [14:52:36] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3224448 (10Marostegui) Some graphs that were shown while troublshooting https://grafa... [14:53:17] 10DBA, 13Patch-For-Review: frwiktionary on s7 still needs fixing on the revision table - https://phabricator.wikimedia.org/T165743#3276513 (10Marostegui) [14:54:28] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276528 (10jcrespo) p:05Low>03Triage This is probably not user-requested invalidat... [15:16:28] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3276584 (10jcrespo) Without entering on heavy rearchitectures, we should, an probably... [15:37:15] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276650 (10jcrespo) [16:01:56] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276828 (10jcrespo) Lots of category pages invalidations happening at that time: ``` UPDATE /* Title::invalidateCache */ `page` SE... [16:04:52] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3276894 (10jcrespo) For the long term: how useful is this field, and could it be separated from the rest of the table if it happens t... [16:08:22] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T165629#3276925 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete. Return information for bad disk attached. {F8123582} [19:56:10] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3277390 (10aaron) The query does not come from HTMLCacheUpdateJob (which calls HTMLCacheUpdateJob::invalidateTitles) or seemingly any... [20:03:54] 10DBA, 06Operations, 06Performance-Team, 10Traffic, 10Wikidata: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3277400 (10aaron) [22:21:03] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 05WMF-NDA: Drop tables with no events in last 90 days. - https://phabricator.wikimedia.org/T161855#3145199 (10Tbayer) @Ottomata: Do we have a list of the 124 tables that were dropped? More generally, the script should log deletions if it doesn't already....