[04:16:46] 10DBA: Re-import some tables on db2094:3318 (sanitarium host) - https://phabricator.wikimedia.org/T282514 (10Marostegui) 05Open→03Resolved This is all done [04:31:16] 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Schema-change, 10User-notice: Mailman3 schema change: change utf8 columns to utf8mb4 - https://phabricator.wikimedia.org/T282621 (10Marostegui) From what I can see the tables aren't huge, so it might not take a long time. However, on the last time we had to alte... [04:40:06] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1112.eqiad.wmnet'] ` The log can be found in... [04:44:02] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), and 2 others: Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas - https://phabricator.wikimedia.org/T104459 (10Marostegui) Very nice @Ladsgroup!!! I have documented t... [04:46:12] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [04:47:11] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:01:05] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1112.eqiad.wmnet'] ` and were **ALL** successful. [05:02:41] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:04:14] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:05:07] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) db1112 reimaged to Buster, - checking tables now. [05:06:06] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:08:56] 10DBA, 10Patch-For-Review: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:11:34] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:15:01] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:17:41] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1079.eqiad.wmnet - https://phabricator.wikimedia.org/T282079 (10Marostegui) [05:24:33] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Kormat) [05:25:20] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [05:26:01] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) [05:26:28] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) All done. Total read-only time: 2m14s. [05:26:47] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10Kormat) [05:26:51] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) 05Open→03Resolved [05:27:12] well done kormat! you are definitely a dba now. You've achieved your childhood dream! [05:27:23] 😭 [05:27:31] you can now add s6 to orchestrator! [05:29:18] * kormat looks up orchestrator on wikitech to remember how any of this works [05:29:25] hahaha [05:30:06] you'd need to add the grants, clean up entry for db1131's serverid on heartbeat table across all the hosts, and then you can discover the topology on orchestrator [05:31:13] ah. but ROW replication will hurt me [05:31:20] i'll add the grants to start with anyway [05:31:50] yeah, for the entry you probably want to set session sql_log_bin=0; and delete the entry per-host rather than from the master [05:32:18] "probably", he says, after having done it a bunch of times :P [05:32:55] hahaha [05:33:31] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1079.eqiad.wmnet - https://phabricator.wikimedia.org/T282079 (10Marostegui) [05:33:52] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1079.eqiad.wmnet - https://phabricator.wikimedia.org/T282079 (10Marostegui) [05:35:23] marostegui: is this expected? [05:35:24] ``` [05:35:28] root@db1173.eqiad.wmnet[heartbeat]> GRANT SELECT ON mysql.slave_master_info TO 'orchestrator'@'208.80.155.103'; [05:35:29] ERROR 1146 (42S02): Table 'mysql.slave_master_info' doesn't exist [05:35:29] ``` [05:35:32] yes [05:35:36] we need to remove that [05:35:43] we don't need it [05:38:43] sigh, right [05:39:02] how does `set session sql_log_bin=0; delete from heartbeat where ts < "2021-05-17T05:30:02.000900"` look? [05:39:28] that might delete codfw entries in codfw too, no? [05:39:40] it'll delete all _stale_ entries, everywhere [05:39:53] that should in theory, leave us with just the relevant ones [05:40:01] I deleted those already, so you only need to do a delete from heartbeat where server_id=db1131's one [05:40:02] at least that's the idea [05:40:07] there should be just 1 row left [05:40:19] db2097:3316, for example, has 3 rows [05:40:36] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Ladsgroup) `wawikisource` in s5 seems not to have it (and it's showing up in drift reports). Can you check s5 again? [05:40:39] jeeeze [05:40:59] so annoying [05:41:01] one from the eqiad primary, one from the codfw one, and the stale one from db1131 [05:41:07] ah right [05:41:11] that makes sense [05:41:20] so you only need to delete db1131's one [05:41:58] so 171974662 is the one that needs tobe delted [05:42:02] *to be deleted [05:42:27] alright. `set session sql_log_bin=0; delete from heartbeat where server_id=171974662 limit 1` [05:42:35] that should work [05:42:37] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission db1079.eqiad.wmnet - https://phabricator.wikimedia.org/T282079 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db1079.eqiad.wmnet` - db1079.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [05:43:53] 10DBA, 10DC-Ops, 10decommission-hardware, 10ops-eqiad: decommission db1079.eqiad.wmnet - https://phabricator.wikimedia.org/T282079 (10Marostegui) a:05Marostegui→03Cmjohnson [05:44:26] 10DBA, 10Commons, 10Patch-For-Review: Avoid capacity issues from the image table holding the text of pdf/djvu files as part of their metadata - https://phabricator.wikimedia.org/T275268 (10Ladsgroup) What you're saying is basically T96384#5234876 and should be done, but that's much further down the road. For... [05:44:28] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [05:47:18] marostegui: so either that was a success, or everything is about to explode [05:47:23] 🤞 [05:47:27] haha [05:48:47] eep. large spike in errors: https://grafana-rw.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=10&orgId=1&var-site=eqiad&var-group=core&var-shard=s6&var-role=All&from=now-30m&to=now [05:50:00] I am not seeing anything on logstash [05:50:27] `Aborted connection 5154877 to db: 'unconnected' user: 'root' host: '10.64.32.25' (Got an error reading communication packets)` [05:50:39] could that be from me running db-replication-tree... [05:50:47] that would be the dumbest thing ever [05:50:57] why would db-replication-tree generate errors? [05:51:11] it seems that we don't cleanly close connections, [05:51:16] and for some reason mariadb takes offense at that [05:51:44] i've noticed this a lot in testing [05:51:52] i just didn't realise it could affect real-life metrics [05:52:02] can you run it again and see if we get another one? [05:52:27] done [05:52:45] new warning in mariadb journalctl log, at least [05:52:50] let's see what the metrics do [05:53:04] there we go the spike [05:53:07] update faster grafana, damn you [05:53:08] https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=10&orgId=1&var-site=eqiad&var-group=core&var-shard=s6&var-role=All&from=now-15m&to=now [05:53:14] yep! [05:53:15] so yeah, that is it [05:53:25] that _definitely_ needs to be fixed [05:53:52] i wonder if 10.1 didn't log a warning for that, but 10.4 does [05:53:59] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) 05Resolved→03Open [05:54:35] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) Will do @Ladsgroup thanks for reporting! [05:57:10] 10DBA: wmfmariadbpy: Close connections gracefully - https://phabricator.wikimedia.org/T282989 (10Kormat) [05:57:28] 10DBA: wmfmariadbpy: Close connections gracefully - https://phabricator.wikimedia.org/T282989 (10Kormat) p:05Triage→03Medium [05:57:48] 10DBA: wmfmariadbpy: Close connections gracefully - https://phabricator.wikimedia.org/T282989 (10Kormat) a:03Kormat [05:58:23] there, sobanski will be so proud of me. i triaged it and put it in the right workboard section 😊 [06:01:14] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) Looks like the new ones miss them, I guess race condition between when the wikis were created and when this was merged or something. Anyways these are the ones... [06:02:31] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) >>! In T270621#7091654, @Marostegui wrote: > Looks like the new ones miss them, I guess race condition between when the wikis were created and when this was me... [06:02:53] marostegui: https://orchestrator.wikimedia.org/web/cluster/alias/s6 \o/ [06:03:32] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [06:03:32] niiiice [06:03:44] !bash there, sobanski will be so proud of me. i triaged it and put it in the right workboard section [06:03:45] Majavah: Stored quip at https://bash.toolforge.org/quip/u4HseHkBa_6PSCT9VjKi [06:03:45] kormat: you are missing clouddb* and sanitarium [06:04:01] kormat: you need to add the grants to those manually (as replication filters prevent them from replicating) [06:04:13] ohh. right. [06:04:20] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Ladsgroup) Thanks. I'm sure this won't happen again for newer wikis (the tables.json have the correct one) You probably can do the change live too. The table is tiny. [06:04:26] so you need to add the grants to eqiad/codfw sanitarium and the clouddb*:3316 [06:04:51] kormat: this that sounds so straightforward took me like 1 hour to figure it out XDD [06:04:55] :D [06:04:59] marostegui: Thanks. You rock! [06:07:39] marostegui: adding the grants to the sanitariums is sufficient. [06:08:05] right [06:08:23] marostegui: what's the story with labsdb*? are they now gone for good [06:08:26] ? [06:08:34] kormat: yeah, they'll be decommissioned this week [06:08:37] \m/ [06:11:23] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) >>! In T270621#7091654, @Marostegui wrote: > Looks like the new ones miss them, I guess race condition between when the wikis were created and when this was me... [06:11:45] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) 05Open→03Resolved [06:14:10] marostegui: https://wikitech.wikimedia.org/wiki/Orchestrator#Adding_a_section_to_orchestrator [06:19:50] oh sweet [06:19:59] thanks [06:27:33] I have added and bolded the fact that we have our first buster + 10.4 master running in a sX section [06:27:37] To today's monday meeting [06:27:39] sobanski ^ [06:37:00] 10DBA, 10MediaWiki-Parser, 10Parsoid, 10Performance-Team: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Marostegui) >>! In T282761#7086572, @dpifke wrote: > Have we considered using table partitioning to make the purge less expensive? I... [09:40:31] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10Kormat) [09:40:47] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10Kormat) [09:42:32] marostegui: thanks! [09:43:12] kormat: with that Phab update of yours, my mission is now complete ;) [09:59:46] I may arrive a few seconds late to the meeting, I got lost checking binary logs [10:00:00] XD [10:00:27] always take a note of the last position before starting so that you don't get lost ;) [10:00:57] or enable GTIDs :-P [11:04:52] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), and 2 others: Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas - https://phabricator.wikimedia.org/T104459 (10LSobanski) A feature request for data differences (narr... [11:09:33] 10DBA, 10SRE: Migrate MySQLs to use ROW-based replication - https://phabricator.wikimedia.org/T109179 (10LSobanski) [11:10:30] 10DBA, 10MediaWiki-General, 10PostgreSQL, 10Schema-change, 10Wikimedia-database-error: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441 (10LSobanski) 05Open→03Resolved a:03LSobanski This task is too large in scope. Tasks for indi... [11:33:37] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [11:33:47] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [11:33:56] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [11:34:13] kormat: can I run 3 schema changes on db1131 (old s6) master before you reimage it? [11:45:08] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10LSobanski) I moved the tracking table to https://wikitech.wikimedia.org/wiki/Obsolete_or_unneeded_database_tables which can be updated inste... [11:45:20] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10LSobanski) 05Open→03Resolved a:03LSobanski [11:45:38] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) <3 [11:49:27] marostegui: be my guest [11:49:38] \o/ [11:49:41] it is half way now XD [11:51:02] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [11:51:16] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [11:51:25] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [11:51:36] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [12:04:43] 10DBA, 10decommission-hardware: decommission db1085.eqiad.wmnet - https://phabricator.wikimedia.org/T282096 (10Marostegui) This is probably good to go now [12:17:44] jynus: is https://gerrit.wikimedia.org/r/c/operations/puppet/+/681622 a dupe of https://gerrit.wikimedia.org/r/c/operations/puppet/+/685494 ? [12:23:51] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) s8 eqiad [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1178 [] db1177 [] db1172 [] db1167 [] db1154 [] db1126 [] db1116 [] db1114... [12:23:54] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) s8 eqiad [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1178 [] db1177 [] db1172 [] db1167 [] db1154 [] db1126... [12:23:58] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) s8 eqiad [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1178 [] db1177 [] db1172 [] db1167 [] db1154 [] db1126 [] db1116 []... [12:24:17] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [12:24:30] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [12:24:42] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [12:25:52] kormat, most likely [12:25:55] let me chec [12:26:11] or maybe I merged the wrong one [12:27:16] it is the same, so I will abandon one [12:28:45] cheers [12:29:20] thanks for pointint out, my WIP gerrit dashboard has 2 pages of patches [12:38:36] Amir1: one of the watchlist schema changes has a datatype change, so it rebuilts the table. On wikidata it went from 11GB to 4.1G (compressed) [12:38:43] so that is nice, as we need to alter all the hsots [12:38:44] hosts [12:39:37] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [12:39:43] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [12:39:50] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [12:47:12] 10DBA: Switchover s6 from db1131 to db1173 - https://phabricator.wikimedia.org/T282124 (10Kormat) [12:59:46] marostegui: oh thanks. It's not that bad. I wish we could squeeze more out of it [13:10:00] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['db1131.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20210517130... [13:36:04] 10DBA, 10Analytics, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10akosiaris) >>! In T120242#7088486, @Ottomata wrote: >> Do we have estimations (or even better ha... [13:57:32] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1131.eqiad.wmnet'] ` and were **ALL** successful. [14:35:21] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10Kormat) db1131 (old s6 eqiad primary instance) has been reimaged to buster. It's now running `mariadb-check -A`. [15:08:25] 10Data-Persistence-Backup: Backup alert email notification - https://phabricator.wikimedia.org/T283017 (10LSobanski) [15:08:51] 10Data-Persistence-Backup: Backup alert email notification - https://phabricator.wikimedia.org/T283017 (10LSobanski) p:05Triage→03Low [15:29:36] 10Data-Persistence-Backup, 10SRE, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10LSobanski) Bonus question, is there an option for some traffic shaping / QoS to remediate the above autom... [15:43:09] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) >>! In T280492#7091567, @Marostegui wrote: > db1112 reimaged to Buster, - checking tables now. db1112 got all the tables checked, everything is c... [15:43:21] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) [17:34:35] 10DBA, 10Patch-For-Review: Upgrade s6 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T280751 (10jcrespo) db2097:s6 removed, only db1139:s6 left (to do next week). [18:07:50] 10DBA, 10MediaWiki-Parser, 10Parsoid, 10Performance-Team: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Krinkle) [18:28:21] 10DBA, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Introduce alerting to monitor mediawiki databases QPS rate of change - https://phabricator.wikimedia.org/T281833 (10Gilles) [19:27:45] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [19:32:23] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [21:45:44] hello! I was told I should share my findings that the new Toolforge replicas are 10-15x as fast as the old ones. Which is a surprise because here I was doing some giant UNION queries to get data across several dbs, and now I have to use different connections and piece the results together in PHP. Yet, somehow, it's still vastly outperforming the old system. So, thank you :) [22:02:55] Hiya, don't know if it means much to you guys, but wanted to thank you (even it's belated) the amazing speed the users are now enjoying with the Wiki Replicas. GUC is so fast now! :) [23:00:15] 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Schema-change, 10User-notice: Mailman3 schema change: change utf8 columns to utf8mb4 - https://phabricator.wikimedia.org/T282621 (10Legoktm) How about 2021-05-19 06:00 UTC? Or any other day at that time Note: I reduced the lists of alters based on discussions w...