[04:56:42] 10DBA, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and 10.4 - https://phabricator.wikimedia.org/T249188 (10Marostegui) Yes [04:58:17] 10DBA, 10Data-Services, 10Quarry: Quarry query became work much slower - https://phabricator.wikimedia.org/T247978 (10Marostegui) Unfortunately, the servers that we use for Quarry and for the all wikireplicas in general is very specific (and very costly) so we do not have hot spares ready to take over any mo... [05:03:14] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Marostegui) a:05jcrespo→03Papaul [05:03:28] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Marostegui) >>! In T251639#6101118, @RobH wrote: > @jcrespo or @Marostegui: > > The racking details from the ordering task only list 4 hosts, but we ended... [05:16:40] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 31st May) rack/setup/install db114[1-9] - https://phabricator.wikimedia.org/T251614 (10Marostegui) [07:08:07] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Prevention), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [07:08:09] 10DBA, 10MediaWiki-User-management, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.35-notes (1.35.0-wmf.30; 2020-04-28), and 2 others: Rename ipb_address index on ipb_address to ipb_address_unique - https://phabricator.wikimedia.org/T250071 (10Marostegui) [07:08:11] 10DBA: inverse_timestamp column exists in text table, it shouldn't - https://phabricator.wikimedia.org/T250063 (10Marostegui) [07:08:13] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [07:08:15] 10DBA: lc_lang_key index is lingering in production - https://phabricator.wikimedia.org/T250056 (10Marostegui) [07:08:17] 10DBA: type_acton index in logging table is lingering in production - https://phabricator.wikimedia.org/T250057 (10Marostegui) [07:08:19] 10DBA: searchindex indexes are missing in production - https://phabricator.wikimedia.org/T250058 (10Marostegui) [07:08:21] 10DBA: db1110 has 5 important database drifts that are unique to the host - https://phabricator.wikimedia.org/T249973 (10Marostegui) [07:08:24] 10DBA, 10wikitech.wikimedia.org: Wikitech has lots of database drifts with core and rest of the databases - https://phabricator.wikimedia.org/T249972 (10Marostegui) [07:25:43] 10DBA, 10Operations: Upgrade and restart s5 and s6 primary DB master: Tue 5th May - https://phabricator.wikimedia.org/T251154 (10Marostegui) 10.1.43-2 has been installed on both masters (without mysql_upgrade) and they are ready for tomorrow's restart. [07:33:36] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Marostegui) [07:34:38] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Marostegui) @Papaul the initial puppet changes are done. From puppet side the only pending thing is; to add them to the DCHP file (if... [07:37:14] 10DBA, 10User-DannyS712: Drop flagged revs tables on mediawikiwiki - https://phabricator.wikimedia.org/T248298 (10Marostegui) 05Open→03Resolved Dropped everywhere. I kept a temporary tiny backup at: ` root@cumin1001:/home/marostegui/T248298# ls -lh flagged* -rw-r--r-- 1 root root 2.7M May 4 07:30 flaggedi... [07:37:19] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [07:39:06] Amir1: I am going to go ahead and drop wb_terms on the labsdb hosts [08:13:52] 10Blocked-on-schema-change, 10DBA: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) a:03Marostegui [08:22:05] 10Blocked-on-schema-change, 10DBA: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) [08:31:22] 10Blocked-on-schema-change, 10DBA: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10jcrespo) Question: are block functionality maintainers (https://www.mediawiki.org/wiki/Developers/Maintainers#MediaWiki_core) aware of these issues, fixes?... [08:31:48] 10Blocked-on-schema-change, 10DBA: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) Heh, this needs to be made in different transactions as we were hit (again) by: https://jira.mariadb.org/browse/MDEV-8351 (I reported it too a f... [08:32:09] 10Blocked-on-schema-change, 10DBA: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) [08:46:44] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) Good question @jcrespo - on the other hand, wikidata (and all the new s3 wikis) are being created with this 4 columns into... [09:16:35] re:netboot.cfg, wouldn't be easier to have db[12]* ? [09:16:53] we can do that now, yes [09:17:33] as long as any exception is defined above should be fine anytime [09:19:53] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10Kormat) I'm going to work on db1001 and es2025. [09:35:58] what's db1001? [09:36:20] a test [09:36:37] like, a vm? [09:36:52] no, like a typo :) fixed now, db1101 [09:36:58] ah, I see [09:37:00] sorry [09:37:06] I was like super-confused [09:37:13] because that db host actually existed [09:37:22] don't be, i'm glad you asked :) i'm paranoid about typing the wrong hostname, so it's good to know when it happens [09:38:00] Re: errors, don't worry [09:38:36] most errors will eventually come a few months in, when withing the gap of confidence :-D [09:39:02] haha [09:58:05] marostegui: cool! Thanks [10:20:55] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['db1101.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202005041020... [10:41:02] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1101.eqiad.wmnet'] ` and were **ALL** successful. [11:07:15] jynus: i have no idea why tendril needs to die - is there a phab task or something that gives context? [11:07:40] yep [11:07:45] one sec [11:10:20] kormat: run! ;) [11:11:14] good point, lunch! [11:40:55] Amir1: everytime I inspect recentchanges queries I really want to cry [11:48:27] I feel you 😭 [11:48:52] there is a task for that [11:51:13] there are multiple [11:51:26] multiple tasks for crying? [11:51:28] :-D [11:52:30] kormat: technically tendril doesn't have to die, just transform into something maintainable and with a modern stack- functionality has to stay the same (or improved) [12:11:33] hmm. prometheus-mysqld-exporter.service is enabled on db1101, but it's a multi-instance host [12:12:00] yeah, that is a bug [12:12:04] that shoude disabled [12:12:38] common package is uses and that enables it by default [12:12:57] which is known to be problematic and cause races between package install & puppet [12:13:04] (in general) [12:18:34] ah, lovely. [12:19:01] it could be still added to puppet, I think on the appropiate roles [12:19:08] as a workaround [12:19:46] there is a few upgrading issues documented at: https://wikitech.wikimedia.org/wiki/MariaDB#Stretch_+_10.1_-%3E_Buster_+_10.4_known_issues [12:20:05] which we don't have enough information about to decide how to solve or workaround [12:21:50] feel free to add that "disable X for multi-instance hosts" [12:21:58] and we can later puppetize it [12:22:11] or repackage, depending on the best way to solve it [12:25:18] done [12:42:55] marostegui: PTAL at the dbctl config diff on cumin1001 for repooling db1101 into s7 and s8 [12:43:10] checking [12:43:11] s7 is straight to 100% (weight 150), s8 is at 50% (weight 175) [12:43:44] I would suggest 25% for s8 and 50% for s7 instead [12:43:52] ah, ok :) [12:43:52] s8 can hit hard [12:45:01] updated [12:45:29] checking [12:46:02] notifications are enabled and the hosts are green on icinga? [12:46:10] correct [12:46:15] then +1! [12:46:24] not +2? :( [12:46:26] (kidding :) [12:46:34] XDD [12:58:51] marostegui: how do i know when it's time to increase the percentage? [12:59:55] kormat: normally 10 minutes are ok, what can give you more indications are the per host graph and if the graphs are stable, specially the number of connections, traffic, innodb buffer pool efficiency and disk latency: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&from=now-15d&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1010&var-port=9104 [13:00:02] that's for labsdb1010 [13:00:33] if you go for db1101, remember you need to change the port as those are multi instance, 9104 isn't going to give you any data, but 1331X where X is the section [13:00:38] so in that case 13311 and 11317 [13:01:17] oh yeah, on that topic - who owns the mysql dashboard? [13:01:35] because it's ~trivial to change the variable config such that only the relevant ports will be shown in the dropdown [13:01:36] I don't think it has a clear owner :) [13:01:44] kormat: go for it! [13:02:21] do we use graffonet or similar to generate the dashboards [13:02:25] or is the live copy the source? [13:02:39] the live copy is the source as far as I know [13:02:43] godog: ^ [13:03:15] it depends [13:03:22] some are defined in puppet, some are "live" [13:03:23] ™ [13:03:36] those defined in puppet IIRC aren't editable in the UI (or at least used to be) [13:03:59] Specifically the mysql one I don't think it is on puppet [13:04:05] mandatory reference: T171482 [13:04:06] T171482: Programmatic generation of grafana dashboards - https://phabricator.wikimedia.org/T171482 [13:04:11] I have changed some items there to adapt it to 10.4 stuff [13:04:20] and [13:04:21] T178690 [13:04:22] T178690: Better organization for SRE grafana dashboards - https://phabricator.wikimedia.org/T178690 [13:09:55] yup, what has been said already basically [13:10:03] is there a way to run queries directly against prom itself? [13:10:14] i'm assuming the prom instances aren't publically accessible [13:10:42] that's correct, we don't expose the prometheus web interface behind ldap even but you can ssh-tunnel to it [13:10:54] searching for the wikitech link [13:11:10] https://wikitech.wikimedia.org/wiki/Prometheus#Access_Prometheus_web_interface [13:11:23] great, thanks [13:12:00] kormat: there is also the Explore feature in Grafana [13:12:03] np! [13:12:16] that gives you quick access to explore metrics, IIRC you have to be logged in [13:12:38] ah. i haven't used that before. i'll give it a shot [13:13:31] volans: this is useful, thanks :) [13:14:23] * volans completed the 1 useful action / day goal :D [13:14:51] marostegui: check this out: https://grafana.wikimedia.org/d/kUxjfbeWk/mysql-kormat?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1101&var-port=13317 [13:15:04] if you change the server dropdown, you'll only get valid ports in the port dropdown [13:15:13] I see, that's nice yeah [13:15:27] Otherwise you have to start guessing if you don't know which sections that host belong to [13:15:35] yep [13:15:49] ok. i'll make the change to the main dashboard [13:15:57] kormat: don't worry, there will be a moment when you know just by looking at the hostname [13:16:00] kormat: `1 [13:16:00] (almost wrote 'db' as a shorthand, then realised that might be confusing :) [13:16:02] +1 [13:21:29] marostegui: db1101 is looking healthy to me [13:23:08] shall i add another 25% to both sections? [13:23:15] sounds good to me yeah [13:23:53] mmm, is it me or I cannot see any data on its dashboard now? [13:24:15] ah nevermind [13:24:17] I am stupid [13:24:24] +1 :-P [13:24:27] don't scare me like that :) [13:24:28] XDDD [13:25:36] marostegui: diff ready on cumin1001 [13:25:50] checking [13:26:29] +1 [13:26:45] * volans wonders if we should add a more integrated cross-check capability to dbctl [13:27:08] what I would love to have is a dbctl restore-config [13:27:09] or something [13:27:18] like to roll back the last change [13:27:37] we have rollback capability for the MW side (context for kormat) [13:27:47] we miss it on the normalized side of things [13:28:13] it is nice that dbctl still provides you the way to restore the last one, but if you have lost all that in your terminal, it can be tricky [13:28:24] ? [13:28:50] marostegui: just make a shell one-liner to itarate over everything in /var/cache/conftool/dbconfig, and hit ctrl-c when you think it restored the right onw. simples. [13:28:59] volans: the output of: Previous configuration saved. To restore it run: dbctl config restore /var/cache/conftool/dbconfig/20200504-075148-marostegui.json [13:29:03] that's nice [13:29:35] marostegui: yeah so just go to /var/cache/conftool/dbconfig and ls | tail [13:30:04] volans: as I was saying, it is nice it can still be done, but I would to have it with just dbctl restore-last or whatever [13:30:23] ahhh, ok, I misunderstood [13:30:36] the current problem is that that would be local to the host, and that sucks [13:30:52] a better approach would be to move everything to a git repo that is automatically replicated between the two hosts [13:30:58] (like we do for the Netbox generated dns) [13:30:59] yeah, I remember we discussed all this during the deployment of dbctl [13:31:24] at that point it should be also doable to save the normalized state and be able to revert that one too [13:31:48] and/or at that point remove the normalized view from etcd entirely [13:31:52] and have it only on git [13:33:57] marostegui: going to kick off reimaging of es2025 now. it looks like the section in general is very lightly loaded, and the host is in codfw, so i'm going to depool it directly. [13:34:17] kormat: yeah, it doesn't get any traffic being in codfw, go for it [13:44:43] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) [13:49:05] 10DBA: Make partman/custom/no-srv-format.cfg work - https://phabricator.wikimedia.org/T251768 (10Kormat) [13:49:29] 10DBA, 10Operations: Make enabling reimaging for db hosts more humane - https://phabricator.wikimedia.org/T251392 (10Kormat) 05Open→03Resolved a:03Kormat Closing this, and opened T251768 to cover fixing the partman recipe. [13:58:09] marostegui: es2025 has a different pxe path than the other hosts: http://apt.wikimedia.org/tftpboot/stretch-installer-bootif/ [13:58:30] kormat: Yeah, that was a fun thing [13:58:32] Let me look for the ticket [13:59:03] kormat: https://phabricator.wikimedia.org/T242481 [13:59:20] I believe this was fixed on Buster, moritzm am I right? [14:06:36] looking [14:06:58] moritzm: I believe the hack was just because they were being installed with stretch [14:07:06] no? [14:07:15] yeah, on buster this works out of the boot, we only need the stretch-bootif setting for these specific servers when installed with stretch [14:07:24] out of the box :-) [14:07:31] sweeeet [14:07:40] moritzm: great, thanks [14:07:46] kormat: you so you can treat them as a normal server now then, just remove the installer line [14:08:00] when all those hosts are moved to buster, ping me and I'll axe the stretch-installer-bootif stuff [14:08:15] Wow, moritzm that issue was BEFORE all hands? [14:08:32] I thought it happened like a month ago only [14:11:11] times flies when you're having a pandemic :-) [14:11:18] haha [14:11:43] yeah, during the all hands we were seeing the covid as a very far away thing, and now look at us [14:11:56] I modified my concept that it's always the same week to one where each week is a new generation of the previous, with tiny differences driven by evolution [14:12:04] wait [14:12:08] where is april? [14:12:15] I think I lost it! [14:12:29] jynus: was an april's fool ;) [14:12:44] no april in a pandemic year [14:13:10] I hope this is the first and last year I have to spent in a pandemic! [14:13:18] *birthday [14:14:06] marostegui: db1101 still looks ok to me. time to increase to 100% s7, 75% s8? [14:14:26] kormat: go for it! [14:15:52] I wrote something for new dbas, let me find it [14:16:11] jynus: was it "Run The Fuck Away" repeated 1000 times? [14:16:23] kormat: https://phabricator.wikimedia.org/P7516 [14:16:27] it may help as a checklist [14:16:34] just not all at the same time [14:16:55] ignore the [DONE] [14:18:02] jynus: I am doing some math about the s4 hosts and all that, and we can definitely have the backup testing host back [14:18:17] after the new hosts, right? [14:18:21] yep [14:18:24] cool [14:18:39] even with redundancy, right? [14:18:51] jynus: Also, db1102 (I believe is a backup sourcE will be replaced with one of the new hosts too) with larger disk [14:18:59] unless you prefer not to (right now it is 50%) [14:19:06] jynus: yep, even with redundancy [14:19:22] wait, what? [14:19:37] is that as a natural, speed up refresh happening now? [14:19:42] or a new thing [14:19:48] jynus: no, that is part of SDC expansion [14:19:49] or happening next year [14:20:48] jynus: the SDC expansion was meant to get new hosts with larger disks for all eqiad hosts needing it, and the backupsource host was included of course [14:20:57] oh [14:21:02] I didn't remember that [14:21:07] surprise! [14:21:17] that's actually great news [14:21:34] I thought that was the broken source [14:22:03] No, but we can do some moves there if you like and send db1102 to replace db1140 [14:22:13] and once db1140 is fixed, we can put it somewhere else as a core host [14:22:18] well, not sure if to replace it [14:22:26] but more disk will give us more flexibility [14:22:47] the problem with backups is that they take 5 minutes more every time [14:22:53] what I am saying is that we can use a new host to replace db1102 (as scheduled) and move db1102 to replace db1140's role [14:23:07] so from 2 years ago to today, it is taken several hours more [14:23:20] yep, I got you [14:23:45] and once db1140 is fixed we can fit it somewhere else within s1,s2,s3 etc [14:24:01] anyways, TLDR; db1102 will be upgraded and the backup testing host will be back :) [14:24:17] thanks kormat for T251392 ! [14:24:17] T251392: Make enabling reimaging for db hosts more humane - https://phabricator.wikimedia.org/T251392 [14:29:29] you are almost welcome! :) [14:38:28] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) For the record, I have caught this query using `tmp_3`: ` explain SELECT /* SpecialRecentChanges::doMainQuery */ /*! STRAIGHT_JOIN *... [14:43:45] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['es2025.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [15:06:30] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10dmaza) IIRC `ipb_anon_only` and `ipb_auto` are mutually exclusive from the business logic perspective and that's why we probably haven'... [15:11:27] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es2025.codfw.wmnet'] ` and were **ALL** successful. [15:11:36] marostegui: db1101 still seems fine. doing the final 25% on s8. [15:11:41] sweet! [15:11:44] thank you [15:16:57] de nada [15:17:30] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10dmaza) Disregard my last comment. I was still organizing my thoughts and pressed submit by mistake. I think we don't need the 4th colu... [15:20:05] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) >>! In T251188#6105052, @dmaza wrote: > Disregard my last comment. I was still organizing my thoughts and pressed submit by... [15:25:30] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10Cmjohnson) a:05Cmjohnson→03Jclark-ctr @Jclark-ctr can you start the process with HPE please. [15:25:32] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Ladsgroup) It might be a covering index though meaning it works better with four columns instead of three. [15:39:59] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10dmaza) **Extra info:** Seems to have been added back in 2009 `git show 4124558d7b4 maintenance/tables.sql`. @Marostegui let me doubl... [15:52:05] marostegui: es2025 done. [15:52:19] Excellent!!! No issues with the 10G cards this time then? [15:52:26] nope, all went smoothly [15:52:34] yay! [15:53:01] 10DBA, 10Operations, 10User-notice: Upgrade and restart s4 (commonswiki) primary database master: Tue 12th May - https://phabricator.wikimedia.org/T251502 (10Trizek-WMF) [16:35:10] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10dbarratt) >>! In T251188#6105052, @dmaza wrote: > I think we don't need the 4th column as part of the index. From the product perspecti... [16:58:00] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Tchanders) @jcrespo @Marostegui - thanks for pinging AHT. This would explain {T46657}, where the example given is duplicate IP blocks w... [17:06:07] 10DBA, 10DC-Ops, 10Operations, 10Sustainability (Incident Prevention): PXE Boot defaults to automatically reimaging (normally destroying os and all filesystemdata) on all servers - https://phabricator.wikimedia.org/T251416 (10jcrespo) [17:15:25] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install backup1002 + array - https://phabricator.wikimedia.org/T250816 (10jcrespo) Thanks, will take it from here, I should be able to handle this on my own unless unexpected issues arise. [18:20:45] 10DBA, 10Core Platform Team, 10MediaWiki-API, 10Wikimedia-production-error: LBFactoryMulti: Unknown cluster 'cluster14' - https://phabricator.wikimedia.org/T251778 (10Reedy) p:05Triage→03Lowest There's no cluster14 in db-eqiad.php... Tagging #dba but imagine they might not be aware of history from 2007... [18:21:36] 10DBA, 10Core Platform Team, 10Wikimedia-General-or-Unknown, 10Wikimedia-production-error: LBFactoryMulti: Unknown cluster 'cluster14' - https://phabricator.wikimedia.org/T251778 (10Reedy) [22:15:14] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Niharika) Would it be possible to tell ahead of time if there are duplicate blocks on our projects already? If du...