[04:20:02] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Bstorm) > > I don't think that there honestly is a Cloud wide use case for these tables until we have a soluti... [09:08:35] 10DBA: dbstore1002 lagged on creating database - https://phabricator.wikimedia.org/T212908 (10Peachey88) [09:40:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:19:22] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:26:57] 10DBA: dbstore1002 lagged on creating database - https://phabricator.wikimedia.org/T212908 (10Marostegui) That's a known issue on dbstore1002 load+tokudb+indexes on the new tables. It always happens with new wiki creations. dbstore1002 and tokudb will go away soon with the new infra, so probably not worth spendi... [10:27:55] 10DBA: dbstore1002 lagged on creating database - https://phabricator.wikimedia.org/T212908 (10Banyek) 05Open→03Resolved a:03Banyek hm, then I close the ticket, thanks @Marostegui [10:38:33] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) +1 on removing these tables as mentioned in T210693 too. [11:31:13] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) Quick meta-point: @Halfak wrote > This conversation is getting very Producty but if this is what TechCom needs to in or... [14:12:33] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) On labsdb1010 this would be the quickest (with depooled host) `#!/bin/bash MYSQL="sudo mysql --skip-ssl... [15:02:32] 10DBA, 10SDC Engineering, 10SDC General, 10Wikidata, and 3 others: Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Marostegui) >>! In T197616#4853038, @Reedy wrote: >>>! In T197616#4853024, @greg wrote: >> Status? > > https://test-common... [15:21:02] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Marostegui) The cause of the crash was apparently memory related ` /admin1/system1/logs1/log1-> show record1 properties CreationTimestamp = 20190103073754.000000-360 ElementNa... [15:21:04] 10DBA, 10Operations, 10ops-codfw: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March - https://phabricator.wikimedia.org/T130702 (10Marostegui) [15:21:07] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Marostegui) [15:24:34] 10DBA, 10Operations, 10Patch-For-Review: es2019 crashed again - https://phabricator.wikimedia.org/T149526 (10Marostegui) [15:24:37] 10DBA, 10Operations, 10ops-codfw: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March - https://phabricator.wikimedia.org/T130702 (10Marostegui) [15:41:11] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Marostegui) >>! In T212154#4849981, @bmansurov wrote: > @Marostegui does 50 sound good to you?... [15:45:05] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2019 is not responsive - https://phabricator.wikimedia.org/T212833 (10Banyek) On Cumin2001 I have a comparison screen running inside of a screen in `/home/banyek` The script is used the following: `#!/bin/bash for db in $(mysql.py -h es2018 -BN -e "... [15:52:18] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Marostegui OK, could you review the [[ https://gerrit.wikimedia.org/r/#/c/operations... [15:53:03] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) [15:55:12] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Marostegui) Thanks! So, we need 12 connections for the service to be even deployed, and what are... [16:01:59] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Marostegui apparently we have 192 service workers according to T212154#4828960 and e... [16:36:55] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) ^ Nevermind that. Was a copy-paste mistake [19:42:30] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T212966 (10Marostegui) a:03Papaul Let's get it replaced when you can Thanks! [19:42:45] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T212966 (10Marostegui) p:05Triage→03Normal [19:44:51] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui)