[04:19:14] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 8.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [04:21:10] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [05:26:23] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) On-going transfers: es2013 -> es2032 es2016 -> es2033 es2019 -> es2034 [06:04:17] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) es2029 fully pooled in es3 es2030 fully pooled in es1 [06:04:19] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) [07:43:36] 10DBA, 10Wikidata, 10Wikidata-Campsite: Investigate indexes of wb_changes - https://phabricator.wikimedia.org/T262856 (10Marostegui) p:05Triage→03Medium a:03Marostegui Yep, I will try to pick db2082 which is the one with the less weight. We can also try to capture some of the heaviest queries and repla... [08:01:32] marostegui: is it ok to start repooling db2125 (T263244) so we can see if the config tweak helps? [08:01:32] T263244: Reimage and reclone db2125 - https://phabricator.wikimedia.org/T263244 [08:17:38] kormat: yep!! [08:18:40] `repool db2125 "reimage+reclone done T263244" 25 50 75 100` running [08:18:41] T263244: Reimage and reclone db2125 - https://phabricator.wikimedia.org/T263244 [08:43:31] kormat: https://gerrit.wikimedia.org/r/c/operations/puppet/+/628611 going to enable notifications on db2125 [08:44:00] marostegui: will it alert for puppet being disabled? [08:44:31] kormat: yeah, let's enable it, it won't change the live setting [08:44:41] ahh, ok. [08:44:42] Going to create a task to track those tests, as I am doing some more live testing [08:44:43] enabling it now. [08:44:48] you merge my change? [08:45:35] will do as soon as a new puppet run completes [08:45:43] excellent [08:45:44] thanks [08:49:56] 10DBA, 10Operations, 10ops-codfw: db2127 memory errors - https://phabricator.wikimedia.org/T262247 (10Marostegui) @Papaul db2127's is now off, you can proceed whenever you want with the upgrades [08:51:02] done x2 [08:51:09] <3 [09:18:24] 10DBA: Clean up DB related pages on Wikitech - https://phabricator.wikimedia.org/T263420 (10LSobanski) [09:19:09] 10DBA: Clean up DB related pages on Wikitech - https://phabricator.wikimedia.org/T263420 (10LSobanski) p:05Triage→03Low [10:51:00] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) > your idea is to store this in external storage? I'm not sure external storage is the right solution here. For not... [10:55:30] sobanski: just to confirm, the idea is to assign everything to kormat no? [10:59:04] marostegui: te odio [10:59:21] in old spanish that means yes, so I guess I will start assigning them [10:59:25] thanks for the quick answer [11:00:10] marostegui: I'll create a task to assign everything to kormat [11:00:17] ...and assign it to kormat [11:00:36] use the user-kormat tag too, to make sure she's aware [11:02:26] 😭 [11:04:29] Have you looked at the manual? Maybe the --help flag is a cry for help. [11:04:46] Wrong channel, ignore ;) [11:19:12] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Marostegui) Thanks for the additional info. Some comments inline >>! In T261411#6479145, @kostajh wrote: >> your idea is to... [11:20:01] 10DBA: Clean up DB related pages on Wikitech - https://phabricator.wikimedia.org/T263420 (10LSobanski) [11:21:35] kormat: https://usercontent.irccloud-cdn.com/file/vdR49KqV/image.png [11:21:49] I can read spanish too! hehe [11:22:13] :D [11:40:31] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) >>! In T261411#6479294, @Marostegui wrote: > Thanks for the additional info. Some comments inline > So that's a stat... [11:47:16] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) This host crashed again: ` -------------------------------------------------------------------------------- SeqNumber = 890... [11:47:24] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Marostegui) 05Resolved→03Open [11:49:18] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 8 others: Restart extension1 (x1) database primary master (db1120) - https://phabricator.wikimedia.org/T250701 (10Elitre) [11:53:53] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) [11:53:59] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) p:05Triage→03Medium [12:28:46] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) db2129 reverted to `all` db2071 s1 slave, set to `inserts` [13:11:13] 10DBA: Create a "how to engage us" process and documentation for Data Persistence - https://phabricator.wikimedia.org/T263456 (10LSobanski) [13:11:39] 10DBA: Create a "how to engage us" process and documentation for Data Persistence - https://phabricator.wikimedia.org/T263456 (10LSobanski) p:05Triage→03Medium [13:21:27] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) db2081 (wikidata) set to `inserts` [14:02:43] ACKNOWLEDGEMENT - MariaDB sustained replica lag on db2117 is CRITICAL: 12 ge 2 Kormat schema change https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2117&var-port=9104 [14:02:48] ACKNOWLEDGEMENT - MariaDB sustained replica lag on db2117 is CRITICAL: 12 ge 2 Kormat schema change https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2117&var-port=9104 [14:03:07] ^ my fault. i forgot to downtime before starting the schema change. [14:03:17] * marostegui silences his phone [14:03:30] it's fiiiiine now [14:03:39] (hopefully :) [14:03:43] ¡ok! [14:09:52] One mystery solved: when you create / edit a project in Phabricator you can choose color :) [14:10:55] Have you found out the difference between for instance #blocked-on-schema-change and #dba? are they essentially the same thing (tags) with just different colours? [14:11:05] Can also choose icon [14:11:32] sobanski: The only thing I see on blocked-on-schema-change vs dba is that dba says group and the other says tag [14:11:51] That is just the icon choice (as far as I see here) [14:12:24] But check this: https://phabricator.wikimedia.org/project/profile/1060/ (group) and https://phabricator.wikimedia.org/project/profile/1494/ (tag) [14:12:29] is that just visual? [14:12:33] or does it have more implications? [14:13:08] https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_Projects [14:13:41] Interesting.... [14:13:45] Group (formerly Team) corresponds to an existing team. If you belong to a group that will manage several projects, then create a Project for your group. Group Projects can be automatically added to (sub)component tasks by requesting a global Herald rule. Icon+Color: Group+Violet. [14:13:49] Tag is used as a cross-component keyword (like "accessibility"), a "never-ending" project. Icon+Color: Tag+Yellow [14:14:59] but I'm quite sure that T2001 applies here too (for sobanski, that's bug #1, why it's 2001 is left an exercise to the reader) [14:15:00] T2001: [DO NOT USE] Documentation is out of date, incomplete (tracking) [superseded by #Documentation] - https://phabricator.wikimedia.org/T2001 [14:15:58] Nice [14:20:39] marostegui: should we close T263244? the reimage/reclone is done. tracking the config tweak is being done in a different task, and the hw failure is the main task [14:20:40] T263244: Reimage and reclone db2125 - https://phabricator.wikimedia.org/T263244 [14:21:07] kormat: +1 yeah [14:21:28] Going to set the config back there lively [14:21:38] 10DBA: Clean up DB related pages on Wikitech - https://phabricator.wikimedia.org/T263420 (10LSobanski) From a conversation with @Ladsgroup "The major migrations definitely can go at least DBAs should do it and the chance of it happening is pretty low the sql.php can and the other ones are good to know but shou... [14:22:07] 10DBA, 10User-Kormat: Reimage and reclone db2125 - https://phabricator.wikimedia.org/T263244 (10Kormat) 05Open→03Resolved Resolving this. Host was reimaged+recloned, but died again due to h/w issues (tracked by {T260670}) [14:22:12] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10Kormat) [14:22:19] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) db2125 (s2) set to `inserts` [14:31:48] 10DBA: Update the task tracking workflow - https://phabricator.wikimedia.org/T263463 (10LSobanski) [14:32:09] 10DBA: Update the task tracking workflow - https://phabricator.wikimedia.org/T263463 (10LSobanski) Requested tag creation permissions in https://phabricator.wikimedia.org/T706. [14:32:26] 10DBA: Update the task tracking workflow - https://phabricator.wikimedia.org/T263463 (10LSobanski) p:05Triage→03Medium [14:33:44] 10DBA: Update the task tracking workflow - https://phabricator.wikimedia.org/T263463 (10LSobanski) [14:42:51] 10DBA, 10MediaWiki-General, 10Patch-For-Review, 10Schema-change: Convert primary key integers and references thereto from int to bigint (unsigned) - https://phabricator.wikimedia.org/T63111 (10Lucas_Werkmeister_WMDE) Another table to keep an eye on is Wikidata’s `wb_changes`. Its `change_id` is thankfully... [15:04:57] 10DBA, 10Operations, 10ops-codfw: db2127 memory errors - https://phabricator.wikimedia.org/T262247 (10Papaul) @Marostegui firmware upgrade complete [15:05:27] 10DBA, 10Operations, 10ops-codfw: db2127 memory errors - https://phabricator.wikimedia.org/T262247 (10Marostegui) Thank you @Papaul - I will take it from here! [15:28:56] 10DBA, 10Privacy Engineering, 10WMF-Legal, 10Performance Issue, and 2 others: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10sbassett) a:03JFishback_WMF [15:32:10] 10DBA, 10User-Kormat: Reimage and reclone db2125 - https://phabricator.wikimedia.org/T263244 (10Marostegui) Tracking for the `innodb_change_buffering` flag is happening at: T263443 [15:36:50] 10DBA: Update the DBA task tracking workflow - https://phabricator.wikimedia.org/T263463 (10Aklapper) [15:41:06] kormat: btw, did you push the repooling script to the repo? [15:43:09] not yet, no [15:43:26] kormat: roger that, worth pushing it! [15:49:39] you just want it entered into evidence [15:51:12] hahaha [15:51:45] kormat: I just want to see your username next to a gerrit patch that contains bash [17:41:41] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) Hello, @Marostegui! We are scheduled to begin the release process this week, as per the schedule. This means that we plan to enable Watchlist Expiry on O... [17:43:04] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) [17:52:05] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Catrope) >>! In T261411#6479345, @kostajh wrote: >> Yeah, I don't think external store would be the place to store this. >> B... [17:54:31] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) [17:55:53] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) [18:24:19] 10DBA, 10DiscussionTools, 10OWC2020, 10Editing-team (FY2020-21 Kanban Board): [SPIKE] What is the process for creating a table to store thread subsriptions? - https://phabricator.wikimedia.org/T260372 (10JTannerWMF) Hey @Joe! I believe you all are the right people to reach out to about tables. Please see t... [22:58:59] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Tgr) There is also the object stash (which on the Wikiemdia cluster resolves to Redis), but I guess we want to use that as li...