[04:56:27] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Marostegui) `testcommonswiki` is done! Thanks for reporting it! [04:56:35] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Marostegui) 05Open→03Resolved [06:39:39] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1092.eqiad.wmnet'] ` The log can be found in `/var/log/wmf... [07:15:48] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1092.eqiad.wmnet'] ` and were **ALL** successful. [07:45:32] 10Blocked-on-schema-change, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat) [07:49:31] 10DBA, 10User-Kormat: Create reuse recipes for tendril/zarcillo/dbprov/backup hosts - https://phabricator.wikimedia.org/T255768 (10Kormat) dbprov recipe was tested yesterday successfully (https://gerrit.wikimedia.org/r/c/operations/puppet/+/618718 + T258749#6363993). [07:49:39] with your permission, I am going to test the viability of creating a memory check on db2102 [07:49:48] +1!!! [07:49:52] that may involve creating a few ooms [07:49:55] heads up [07:50:25] (notifications are disabled, but still, things could go wrong in several ways) [08:21:25] https://gerrit.wikimedia.org/r/c/operations/puppet/+/618947 [08:48:51] Amir1: Can you give your drifts script a go? :) [09:16:00] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=MariaDB+memory I will not try to break db2102 [09:16:03] *now [09:17:38] * marostegui runs [09:22:53] marostegui: Will do, actually I have some good/bad news: Now more than 25% of core tables are migrated to abstract schema which makes the schema drift code for those much easier (you just load the json instead of parsing SQL, believe me I wrote a sql parser... https://github.com/Ladsgroup/db-analyzor-tools/blob/master/db_drift_checker.py#L104). Last night I finished the part for abstract schema and it works just fine which means we [09:22:54] can improve the checks to do size of the field or its data type, and other complicated stuff it's not currently checking which means you're going to have HUGE amount of more database drift on your plate [09:23:54] Amir1: that's cool, it is good to at least know them, but for now I am more worried about normal indexes, unique ones, columns, PKs and all that [09:24:51] aha, sure, it has a flag for those "--important-only" [09:25:07] I will run it later today and give you the result [09:25:07] once we're done with those, we can track the others I think [09:25:13] thank you [09:25:17] indeed [10:32:48] I remember first time that I ran it, it reported 3000 drifts and it was only on s1 and s8 [10:34:19] hahaha [10:34:26] s3 is always there waiting for us [10:38:57] the big problem is that if I check all wikis, any sort of drift will overwhelm the report but picking one wiki per section turns it to a whack-a-mole game :( [10:39:28] specially since it picks up the wiki at random so different runs yield different results [10:43:26] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), and 3 others: Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Naike) [10:58:03] 10Blocked-on-schema-change, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Marostegui) [11:03:44] to add more issues to tendril, sometimes idp and tendril end up in an infinite redirection look [11:04:01] oh, never happened to me [11:04:13] No, actually it did, weeks ago [11:04:19] I had to clear cookies and all that [11:04:36] in fact, I think it is reproduceble [11:04:54] can you load https://tendril.wikimedia.org/host/view/db2102.codfw.wmnet/3306 ? [11:05:00] yep, works [11:05:05] so it is only me [11:05:14] yeah, when it happened to me, it was just me [11:05:16] clear all cookies [11:05:19] that worked for me [11:07:42] I cleared ipd and tendril cookies and then restarted the browser [11:08:33] and no luck? [11:08:42] yeah, it worked after that [11:08:56] I didn't want to clear all cookies, too much things to relogin [11:09:01] *many [11:12:47] hehe yeah [16:50:24] 10DBA, 10User-Urbanecm: Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Urbanecm) Well, anything that works for you works for me too, I really shouldn't be the blocking one. I thought what happened now could happen at any point in the future. If you're saying... [17:05:02] 10DBA, 10User-Urbanecm: Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Marostegui) Yeah, let's change it once we've moved the two wikis on Tuesday. Then we can update the documentation and all that. Thank you very much for your help [17:28:09] 10DBA, 10observability, 10Sustainability (Incident Followup): Monitor swap/memory usage on databases - https://phabricator.wikimedia.org/T172490 (10jcrespo) I missconfigured db1121 with 430GB of bufffer pool and started 2 mysqldump processes to have some activity. The alert went off as expected: {F31984764}... [18:50:57] marostegui: sorry, I broke the db reporter today while adding the abstract schema, it should be fixed now [18:51:27] so far looks good now