[06:13:18] 10DBA: Consider setting innodb_adaptive_hash_index=OFF by default - https://phabricator.wikimedia.org/T268869 (10Marostegui) [06:13:30] 10DBA: Consider setting innodb_adaptive_hash_index=OFF by default - https://phabricator.wikimedia.org/T268869 (10Marostegui) p:05Triage→03Medium [06:22:48] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1015.eqiad.wmnet - https://phabricator.wikimedia.org/T268810 (10Marostegui) [06:30:02] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1016.eqiad.wmnet - https://phabricator.wikimedia.org/T268812 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `es1016.eqiad.wmnet` - es1016.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [06:33:17] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1016.eqiad.wmnet - https://phabricator.wikimedia.org/T268812 (10Marostegui) [06:33:47] 10DBA, 10DC-Ops, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission es1016.eqiad.wmnet - https://phabricator.wikimedia.org/T268812 (10Marostegui) a:05Marostegui→03wiki_willy This is ready for #dc-ops [06:39:28] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1015.eqiad.wmnet - https://phabricator.wikimedia.org/T268810 (10Marostegui) [07:00:54] 10Blocked-on-schema-change, 10DBA: Schema change for renaming namespace_title index on watchlist - https://phabricator.wikimedia.org/T268004 (10Marostegui) [07:00:56] 10DBA, 10Orchestrator, 10User-Kormat: orchestrator: integrate promotion rules into puppet - https://phabricator.wikimedia.org/T266002 (10Marostegui) [08:48:35] 10DBA, 10Orchestrator, 10User-Kormat: orchestrator: integrate promotion rules into puppet - https://phabricator.wikimedia.org/T266002 (10Marostegui) To refine a bit on this. We can probably set rules as: `prefer`: Pick the already designed candidate masters `prefer_not`: Sanitarium masters (as they run ROW)... [08:49:46] 10DBA, 10Orchestrator, 10User-Kormat: orchestrator: integrate promotion rules into puppet - https://phabricator.wikimedia.org/T266002 (10Marostegui) [09:39:22] I was checking the new kibana installation and stumbled upon a new db-error: GrowthExperiments\NewcomerTasks\TemplateFilter::buildResultsMap FYI [10:14:39] jynus: that's not new unfortunately: https://phabricator.wikimedia.org/T267216 [10:15:19] TL;DR https://jira.mariadb.org/browse/MDEV-24137 [10:22:03] ih+ [10:22:06] *oh [10:28:03] 10DBA: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10kostajh) >>! In T267214#6651446, @Marostegui wrote: > @kostajh - reminder we are still waiting on knowing from where this database will be accessed. > I could grant 10.64.% or whatever, but if t... [10:42:08] sorry to intervene, on^ but this is me speaking as a backup's person [10:42:30] did you mention the option of providing 2 users, an application and an admin one? [10:42:47] in most cases drop may be needed by admin account, but not app account [10:43:01] this prevents lots of headaches later :-) [10:43:12] and backup recoveries :-) [10:43:18] that was discussed during the meeting, but so far it is not clear [10:43:37] ok, so you brought it up [10:43:43] then, no issue [10:44:09] saying because it is a model we use for other services on misc [10:47:07] if you want me to help with discussion, let me know, for now I will let you handle [10:47:47] also when you create the dbs, I can take care of the db grants if you prefer it [10:47:56] the db grants for backups, I mean [10:48:08] whatever is easier for you [10:48:15] ok, thanks [11:13:36] 10DBA: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10JMeybohm) >>! In T267214#6652689, @kostajh wrote: >>>! In T267214#6651446, @Marostegui wrote: >> @kostajh - reminder we are still waiting on knowing from where this database will be accessed. >>... [11:59:53] 10DBA: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10kostajh) [12:48:07] 10DBA, 10User-Kormat: Add alert for prometheus-mysql-exporter failing to scrape mysql - https://phabricator.wikimedia.org/T257056 (10Marostegui) [13:06:00] 10DBA: Consider setting innodb_adaptive_hash_index=OFF by default - https://phabricator.wikimedia.org/T268869 (10Johnyma22) Related to Etherpad. https://github.com/ether/etherpad-lite/issues/4522 is our link back. You guys have better DBAs than us so will allow your DBAs to feed in. [13:30:26] 10DBA, 10decommission-hardware: decommission es1015.eqiad.wmnet - https://phabricator.wikimedia.org/T268810 (10Marostegui) [13:30:34] 10DBA, 10decommission-hardware: decommission es1017.eqiad.wmnet - https://phabricator.wikimedia.org/T268825 (10Marostegui) [13:30:42] 10DBA, 10decommission-hardware: decommission es1013.eqiad.wmnet - https://phabricator.wikimedia.org/T268436 (10Marostegui) [14:42:03] 10DBA, 10Orchestrator: Configure mariadb to notice/recover from replication issues quicker - https://phabricator.wikimedia.org/T268320 (10Marostegui) I would say let's start by making `slave_net_timeout` consistent across the stack, as before 10.2.3 the default was 3600 seconds and after 10.2.4 it is 60 second... [17:04:01] sorry, but I need to rant- apparently public image containers are hashed in base 16 and deleted image containers are hashed in base 36