[05:23:44] <wikibugs>	 10DBA, 10Schema-change: Add img_deleted column - https://phabricator.wikimedia.org/T257222 (10Marostegui) Removing #blocked-on-schema-change as this schema change isn't really blocked (there is no merged patch on `tables.sql` with the new column). Once we are finally ready for it we can add it back.
[05:58:57] <wikibugs>	 10DBA: Convert Tendril TokuDB tables to InnoDB - https://phabricator.wikimedia.org/T249085 (10Marostegui) I was digging into some tables in tendril, I noticed `query_footprint_log` is empty and running TokuDB, so I have converted it to InnoDB.  ` root@db1115.eqiad.wmnet[information_schema]> select table_schema,t...
[06:06:27] <wikibugs>	 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui)
[07:38:46] <wikibugs>	 10DBA, 10Data-Persistence: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) >>! In T263443#6529382, @Marostegui wrote: > I have noticed some slight increase on InnoDB wait time on pc2009, it could be just a coincidence but I am going...
[07:40:38] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Marostegui)
[08:13:14] <kormat>	 marostegui: dbstore1005 has a memory usage warning: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=dbstore1005
[08:14:01] <marostegui>	 yeah, I decreased dbstore1005's memory a few days ago, maybe I should do the same with this one
[08:14:08] <marostegui>	 sorry, I did dbstore1004
[08:15:07] <marostegui>	 What about this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/633142 ?
[08:15:09] <marostegui>	 elukey: ^
[08:18:01] <elukey>	 thanks a lot, +1ed
[08:18:06] <elukey>	 you can restart anytie
[08:18:08] <elukey>	 anytime
[08:19:17] * kormat creates cronjob to restart it every minute
[08:20:14] <marostegui>	 thanks elukey!
[08:23:34] <kormat>	 marostegui: the s8 schema change in eqiad managed to replicate over night. i'll remove the downtimes
[08:23:43] <marostegui>	 kormat: \o/
[09:20:31] * sobanski heading to a dentist, wish me luck
[09:20:56] <kormat>	 🦷 ⚔️
[09:21:00] <marostegui>	 sobanski: going to the dentist on a friday...I see you like your weekends
[09:22:26] <kormat>	 marostegui: i reckon he just wants the weekend to feel as bad as the week, for consistency. ;)
[09:23:07] <marostegui>	 we like consistency on this team
[09:23:10] <marostegui>	 so +1
[09:23:37] <kormat>	 marostegui: that's an excellent point. :D
[09:26:22] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Persistence, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat)
[09:27:26] <kormat>	 marostegui: what do you think about me kicking off the schema change for s3/eqiad? it's a friday, but the cluster shouldn't alert. and it's probably going to take a while.
[09:27:47] <marostegui>	 kormat: I would play it safe and not do it on a friday
[09:27:58] <kormat>	 alright :)
[09:28:16] <marostegui>	 s3 has too many wikis that it is a bit unpredictable
[09:28:50] <kormat>	 marostegui: alright - how about 1 replica instead?
[09:28:58] <marostegui>	 +1
[09:29:03] <kormat>	 2 replicas? ok!
[09:29:11] <marostegui>	 haha
[09:29:24] <marostegui>	 1 replica is fine to see how long it takes and all that
[09:29:29] <marostegui>	 the table there should be tiny though
[09:30:59] <kormat>	 4G total across all of them
[09:31:35] <marostegui>	 let's see how long a replica takes
[09:31:45] <marostegui>	 maybe it will be done in a couple of hours
[09:55:16] <marostegui>	 kormat: did the change replicate to labsdb finely?
[09:56:30] <kormat>	 possibly.
[09:57:06] <kormat>	 marostegui: yes to 3 of them, labsdb1010 not yet
[09:57:14] <marostegui>	 cooool
[09:57:27] <marostegui>	 and the replica that got the change already is ok too, right?
[09:58:17] <kormat>	 yep, no issues there.
[09:58:27] <marostegui>	 excellent
[09:59:01] <kormat>	 marostegui: running the schema change against dbstore1004:3313 took.. 24mins.
[09:59:20] <marostegui>	 sweeeeeet
[12:51:34] <wikibugs>	 10DBA, 10GlobalUsage, 10Platform Engineering, 10Platform Engineering Roadmap Decision Making, 10StructuredDataOnCommons: Normalize globalimagelinks table - https://phabricator.wikimedia.org/T241053 (10Naike)
[13:05:22] <wikibugs>	 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) @Joe @Marostegui could you please let us know your thoughts on this (T261411#6497293) sometime in the next week?
[13:17:31] <wikibugs>	 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Marostegui) >>! In T261411#6497400, @kostajh wrote: > @MGerlach and I spoke on Friday and he will adjust the output of his to...
[13:45:56] <wikibugs>	 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) > Are we still talking about 18-20k rows in total for this table here as you mentioned at T261411#6479145  If we use...
[13:54:17] <wikibugs>	 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Marostegui) >>! In T261411#6532492, @kostajh wrote: >> Are we still talking about 18-20k rows in total for this table here as...