[00:03:55] PROBLEM - MariaDB sustained replica lag on db1109 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1109&var-port=9104 [00:06:11] RECOVERY - MariaDB sustained replica lag on db1109 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1109&var-port=9104 [06:19:28] 10DBA, 10Growth-Team, 10Notifications, 10Wikimedia-production-error: labswiki: Field 'notification_bundle_display_hash' doesn't have a default value (10.64.0.98) - https://phabricator.wikimedia.org/T262033 (10Marostegui) 05Open→03Resolved [06:21:02] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) Altered db1096:3316 and db1098:3316 in eqiad, if nothing breaks during the next few days I will proceed [06:32:20] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1092.eqiad.wmnet - https://phabricator.wikimedia.org/T275019 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db1092.eqiad.wmnet` - db1092.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [06:35:46] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [06:55:30] 10DBA, 10Patch-For-Review: Reimage db1134 to Buster and repool it - https://phabricator.wikimedia.org/T275343 (10Marostegui) I have started to slowly repool this host [07:12:10] db2094 expected? [07:12:22] yes, see -operations [07:12:33] ok [07:48:34] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1168 is now slowly being pooled into s6 running 10.4.18 [07:48:45] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [07:49:44] 10DBA, 10decommission-hardware: decommission db1088.eqiad.wmnet - https://phabricator.wikimedia.org/T276025 (10Marostegui) [07:50:12] 10DBA, 10decommission-hardware: decommission db1088.eqiad.wmnet - https://phabricator.wikimedia.org/T276025 (10Marostegui) Not yet, let's give db1168 (its replacement a few days to make sure it runs fine) [07:51:29] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [07:51:34] 10DBA, 10decommission-hardware: decommission db1088.eqiad.wmnet - https://phabricator.wikimedia.org/T276025 (10Marostegui) [07:51:38] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [09:25:48] 10DBA: Reimage db1134 to Buster and repool it - https://phabricator.wikimedia.org/T275343 (10Marostegui) [09:26:05] 10DBA: Reimage db1134 to Buster and repool it - https://phabricator.wikimedia.org/T275343 (10Marostegui) 05Open→03Resolved Host fully pooled. [11:42:25] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite, and 5 others: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10LSobanski) [12:55:28] 10DBA, 10Data-Services: Prepare and check storage layer for taywiki - https://phabricator.wikimedia.org/T275836 (10LSobanski) p:05Triage→03Medium Thanks, let us know when the database is created, so we can sanitize it. [13:48:12] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [14:10:43] 10DBA, 10DC-Ops, 10SRE, 10ops-codfw: (Need By: TBD) rack/setup/install db21[45-52] - https://phabricator.wikimedia.org/T273568 (10LSobanski) @Papaul Can it be added to the template or does it need to be added manually to every task? [15:15:06] 10DBA, 10DC-Ops, 10SRE, 10ops-codfw: (Need By: TBD) rack/setup/install db21[45-52] - https://phabricator.wikimedia.org/T273568 (10Papaul) @LSobanski yes it can be added to the template. [17:45:54] 10Data-Persistence-Backup, 10Analytics-Clusters: Evaluate the need to generate and maintain zookeeper backups - https://phabricator.wikimedia.org/T274808 (10fdans) [19:21:29] 10Blocked-on-schema-change: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Ladsgroup) [19:22:02] 10Blocked-on-schema-change: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Ladsgroup) Note: I have two other schema changes for recentchanges table. Will create tickets for those soon. [19:51:33] 10Blocked-on-schema-change: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Marostegui) I will group this one with the other two. But I will start this schema change as soon as possible as I want to get rid of the nightmares of `rc_id` being signed :) [19:52:22] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Marostegui) [19:52:31] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Ladsgroup) {meme, src=itshappening} [19:52:37] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Marostegui) p:05Triage→03Medium a:03Marostegui [19:53:58] 10Blocked-on-schema-change: Drop default of rc_timestamp - https://phabricator.wikimedia.org/T276156 (10Ladsgroup) [19:54:53] 10Blocked-on-schema-change, 10DBA: Drop default of rc_timestamp - https://phabricator.wikimedia.org/T276156 (10Marostegui) p:05Triage→03Medium a:03Marostegui [19:55:12] 10Blocked-on-schema-change: Schema change for dropping default of img_timestamp - https://phabricator.wikimedia.org/T273360 (10Ladsgroup) [19:55:36] 10Blocked-on-schema-change: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Ladsgroup) [20:05:16] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Ladsgroup) While making changes like this are usually harmless but we might end up with errors like {T274091} again. [20:25:16] 10DBA, 10SRE, 10Performance-Team (Radar), 10Sustainability (MediaWiki-MultiDC): Apache <=> mariadb SSL/TLS for cross-datacenter writes - https://phabricator.wikimedia.org/T134809 (10Krinkle) [20:25:47] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Marostegui) Yeah, I will keep an eye on it. I will probably start with s6 (frwiki, jawiki and ruwiki). [21:17:12] 10DBA, 10Data-Services: Prepare and check storage layer for mnwwiktionary - https://phabricator.wikimedia.org/T276126 (10LSobanski) p:05Triage→03Medium Thanks, let us know when the database is created, so we can sanitize it.