[00:05:19] 10DBA, 10Data-Services, 10Chinese-Sites, 10cloud-services-team (Kanban): Prepare and check storage layer for zhwikiversity - https://phabricator.wikimedia.org/T199599 (10Urbanecm) [00:05:25] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for satwiki - https://phabricator.wikimedia.org/T198401 (10Urbanecm) [00:05:29] 10DBA, 10Cloud-Services: Prepare and check storage layer for sahwikiquote - https://phabricator.wikimedia.org/T196362 (10Urbanecm) [00:05:33] 10DBA, 10Cloud-Services: Prepare and check storage layer for pswikivoyage - https://phabricator.wikimedia.org/T196359 (10Urbanecm) [00:05:39] 10DBA, 10Cloud-Services: Prepare and check storage layer for pmswikisource - https://phabricator.wikimedia.org/T195008 (10Urbanecm) [00:05:47] 10DBA, 10Cloud-Services, 10Patch-For-Review: Prepare and check storage layer for idwikimedia - https://phabricator.wikimedia.org/T193187 (10Urbanecm) [05:21:39] 10DBA: Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10Marostegui) With "record" you mean a new column on the `user_properties` table or a new row? The table right now looks like: ` CREATE TABLE `user_properties` ( `up_user` int(10) unsigne... [05:57:45] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [05:58:11] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [06:01:20] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [06:01:47] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) p:05Triage→03Normal a:03Marostegui [06:40:06] 10DBA: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10Marostegui) [06:40:39] 10DBA, 10Patch-For-Review: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10Marostegui) p:05Triage→03Normal [07:03:52] 10DBA: db1115 (tendril) paged twice in 24h due to OOM - https://phabricator.wikimedia.org/T231165 (10Marostegui) p:05Triage→03Normal This looks like the recurrent memory issues tendril has: T196726 however this time it doesn't HW issues [07:03:52] 10DBA, 10Patch-For-Review: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10Marostegui) Also, I am even more forward to keep disabling stuff we don't re... [08:35:38] 10DBA, 10Patch-For-Review: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10jcrespo) +1 [08:36:38] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui)  @cdanis I understand this is related to replicas for main traffic? as in not on specific groups? (ie: api). [09:19:59] 10DBA: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [09:20:12] 10DBA: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [09:20:15] 10DBA, 10Patch-For-Review: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10Marostegui) [09:20:25] 10DBA: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) p:05Triage→03Normal [09:20:46] 10DBA, 10Operations: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [09:20:58] 10DBA: db1115 (tendril) paged twice in 24h due to OOM - https://phabricator.wikimedia.org/T231165 (10Marostegui) [09:21:00] 10DBA, 10Operations: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [09:40:29] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10Marostegui) [09:40:43] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10Marostegui) p:05Triage→03Normal [09:42:21] 10DBA, 10Patch-For-Review: Tendril activity column: trx_adaptive_hash_latched column removed on 10.2 (and onwards) from information_schema.innodb_trx causes the `*_activity` event to fail - https://phabricator.wikimedia.org/T231182 (10Marostegui) 05Open→03Resolved a:03Marostegui I have merged the change,... [09:42:23] 10DBA, 10Operations: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [11:07:33] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10fgiunchedi) Interesting! Since on buster there's an implicit upgrade of mysqld-exporter to 0.11, some of the innodb-related performance schema options... [11:09:11] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10fgiunchedi) re: "monitoring queries latency" the expression needs to be changed like this (i.e. to handle multiple handlers) ` http_request_duration_... [12:10:29] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10Marostegui) The innodb variables, as kinda expected, are failing due to the fact that they've been removed upstream: https://jira.mariadb.org/browse/M... [12:11:15] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10Marostegui) [12:25:39] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10CDanis) That's right -- this just enforces total number of replicas pooled at the top level of a section. We could add a min_pooled value for groups if it would be helpful. [12:26:26] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) >>! In T231019#5437528, @CDanis wrote: > That's right -- this just enforces total number of replicas pooled at the top level of a section. > > We could add a min_pooled value for groups... [12:55:41] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10fgiunchedi) >>! In T231190#5437489, @Marostegui wrote: > The innodb variables, as kinda expected, are failing due to the fact that they've been remove... [13:32:40] 10DBA, 10Operations, 10observability: Investigate with Prometheus doesn't report on some graphs on MariaDB 10.3 - https://phabricator.wikimedia.org/T231190 (10Marostegui) 05Open→03Resolved a:03Marostegui Thanks @fgiunchedi for the explanation and guidance to get it changed. I have replaced it on the da... [13:35:58] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T231199 (10Marostegui) p:05Triage→03Normal a:03Cmjohnson Can we get this disk replaced? This is m1 master. And old host that will get decommissioned soonish (I need to schedule a master failover for it), but a... [13:50:25] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) I have renamed this table on `db2112.enwiki` and will leave it for a few hours to make sure nothing writes t... [13:51:04] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) [14:39:18] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) p:05Triage→03Normal [14:40:24] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) a:03Marostegui I will start taking care of this once eqiad and codfw sections are equal in HW, which is basically once T230106 is fully done. Some sections have already been done, ther... [15:00:09] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s5 done - set it to 3: ` root@cumin1001:~# dbctl -s codfw section s5 get { "s5": { "master": "db2123", "min_replicas": 3, "readonly": false, "ro_reaso... [15:00:55] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [17:07:20] marostegui: I cannot find any new wiki tickets waiting for Cloud. I only see one waiting for upstream to create wiki. Is that right? I thought there was something waiting, but all the emails I got were just for tag changes on closed tasks. [17:12:19] I maybe can fine the one for you [17:13:30] bstorm_: https://phabricator.wikimedia.org/T210762#5421384 [17:14:04] Thanks! That didn't come up in my search for some strange reason [17:14:06] There was https://phabricator.wikimedia.org/T230485 but we merged it, but you can create your own workflows [17:14:13] that one yep, thanks jaime [17:14:22] is there other? [17:14:52] no, just napwikisource [17:15:02] ok, bye [17:35:06] 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Bstorm) a:03Bstorm [17:50:15] 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Bstorm) Created the database and the grant on the replicas, running scripts now to get it all set. [17:55:19] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ayounsi) [17:58:49] 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Bstorm) 05Open→03Resolved Scripts finished. Validated the the views are reachable in Toolforge.