[04:59:05] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10Patch-For-Review: ipb_address_unique has an extra column in the code but not in production - https://phabricator.wikimedia.org/T251188 (10Marostegui) Thank you - I am going to rename the task as the task is no longer to add a column... [05:16:12] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10Patch-For-Review: ipb_address_unique has an extra column in production but not in the code (WAS: ipb_address_unique has an extra column in the code but not in production) - https://phabricator.wikimedia.org/T251188 (10Marostegui) [05:17:09] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10Patch-For-Review: ipb_address_unique has an extra column in production but not in the code (WAS: ipb_address_unique has an extra column in the code but not in production) - https://phabricator.wikimedia.org/T251188 (10Marostegui) I h... [07:17:37] ^so the above case is not an exception, I was told and checked that in the old days, some changes were applied to production with no trace [07:17:52] or maybe as a test but later forgotten [07:18:31] hopefully all are fixed soon [07:19:50] 10DBA: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10RhinosF1) [07:22:30] marostegui: es5 looks nice and calm, so i'm going to remove the weight from the master. [07:22:47] es1024 is fully pooled again since yesterday evening [07:23:42] 10DBA, 10Data-Services: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10Majavah) [07:25:50] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) [07:26:05] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) p:05Triage→03Medium [07:26:29] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) [07:26:31] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) [07:28:13] 10DBA, 10Data-Services: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10ArielGlenn) On which host(s) are you running the above queries? [07:30:28] 10DBA, 10Data-Services: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10Marostegui) You cannot use FORCE (or USE or IGNORE) as those are views and not really tables. [07:32:07] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) [07:32:11] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [07:32:56] 10DBA: Check why compare.py doesn't work with Percona 8.0 - https://phabricator.wikimedia.org/T243265 (10jcrespo) [07:32:58] 10DBA, 10Operations: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10jcrespo) [07:33:08] 10DBA, 10Sustainability (Incident Prevention): Implement (or refactor) a script to move slaves when the master is not available - https://phabricator.wikimedia.org/T196366 (10jcrespo) [07:33:10] 10DBA, 10Operations, 10observability: Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master - https://phabricator.wikimedia.org/T141968 (10jcrespo) [07:33:13] 10DBA, 10Release-Engineering-Team-TODO, 10Epic, 10Release-Engineering-Team (Deployment services): Implement a system to automatically deploy schema changes without needing DBA intervention - https://phabricator.wikimedia.org/T121857 (10jcrespo) [07:34:45] 10DBA: Upgrade and restart s1 (enwiki) primary database master: Thu 21th May - https://phabricator.wikimedia.org/T251982 (10Marostegui) [07:35:06] 10DBA: Upgrade and restart s1 (enwiki) primary database master: Thu 21th May - https://phabricator.wikimedia.org/T251982 (10Marostegui) p:05Triage→03Medium [07:35:46] 10DBA, 10Operations, 10Traffic, 10Patch-For-Review: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462 (10jcrespo) [07:38:47] 10DBA, 10Data-Services: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10Akeron) I used https://quarry.wmflabs.org to test those queries on enwiki_p. It is very penalizing to not be able to force the use of indexes. What is the... [07:39:29] 10DBA, 10Data-Services, 10Quarry: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10RhinosF1) [07:40:57] 10DBA, 10Data-Services, 10Quarry: Unable to use force index on replicas (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10Marostegui) >>! In T251980#6111712, @Akeron wrote: > I used https://quarry.wmflabs.org to test those queries on enwiki_p. > > It is very penali... [07:42:10] 10DBA: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) - https://phabricator.wikimedia.org/T248086 (10Marostegui) p:05Triage→03Medium [07:43:47] marostegui: minor comment left on https://gerrit.wikimedia.org/r/c/operations/puppet/+/594616 [07:44:36] labsdb1011 is on huwiki right now [07:45:16] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) [07:46:59] 10DBA: Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May - https://phabricator.wikimedia.org/T251981 (10Marostegui) [07:50:17] 10DBA: Upgrade and restart s1 (enwiki) primary database master: Thu 21th May - https://phabricator.wikimedia.org/T251982 (10Marostegui) [07:50:27] 10DBA: Upgrade and restart s1 (enwiki) primary database master: Thu 21th May - https://phabricator.wikimedia.org/T251982 (10Marostegui) [07:56:00] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: 31st May) rack/setup/install db114[1-9] - https://phabricator.wikimedia.org/T251614 (10Marostegui) [08:01:45] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: 31st May) rack/setup/install db114[1-9] - https://phabricator.wikimedia.org/T251614 (10Marostegui) Puppet done to get them as spare. As I said, I can do the install myself, what is pending from DCOPs side (apart from racking and ca... [09:11:48] "we make GTID the default on server, but don't support it in our own tools" https://www.reddit.com/r/mariadb/comments/gd79y7/mariadb_point_in_time_recovery_with_gtid/fplwl17/?context=8&depth=9 [09:12:07] not that mysql is any better, but come on! [09:13:24] doesn't surprise me :( [09:13:34] I have backup1002 ready to install [09:13:46] but I am still handling the fallout of tendril yesterday on backups [09:16:03] kormat: I just added you to a meeting someone had forgotten to add you :-( [09:17:57] 10DBA, 10Operations: Upgrade and restart s3 and s7 primary DB master: Thu 7th May - https://phabricator.wikimedia.org/T251158 (10Marostegui) Package upgraded on db1123 and db1086. [09:19:58] marostegui: did you restart tendril db today early in the morning? [09:20:03] no [09:20:26] oh, sorry, my tendril was outdated [09:20:30] it was yesterday's restart [09:20:32] XD [09:20:45] And now you have to login back in! [09:20:59] yeah [09:21:02] exactly [09:21:19] but I was able to kinda-browse in cache for a while [09:21:45] I saw a big drop in traffic on enwiki "a few hours ago" [09:22:00] and thought either outage or monitoring outage [09:22:12] but it was just yesterday's issue [09:40:06] marostegui: for you: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594679 [09:43:25] jynus with the intercept! [09:45:17] I made a comment [09:47:17] replied [09:53:28] after all of these we should have better names, something in the lines of "db-wipe-all-disks" and "db-reimage-keep-srv" whatever, but clearer [09:54:03] agreed [10:07:12] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) 05Open→03Stalled Only s4 master is pending: T250055#6088674 But given how busy and big the `image` table is on s4, I am not going to attempt to do this high risk operation unti... [10:07:19] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Prevention), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [10:08:03] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [10:09:58] marostegui: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594683 (this time i'll wait for your feedback :) [10:10:45] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [10:11:22] I am more of an approach of "if it is not harmful, +1, improvement can happen on following patches" [10:13:08] more of this https://jynus.com/gif/gerrit_plus_1.gifv [10:13:23] and less of this https://jynus.com/gif/i_thought_about_it_and_I_think_it_is_going_to_be_a_minus_1.gifv :-D [10:13:40] hahah [10:14:53] testing this with `touch /srv/pls_dont_leave_me.txt` [10:15:03] and then seeing if it's still there after a reimage :) [10:15:42] cool [10:16:21] there is a chance the fix was trivial, we just didn't see it at the time [10:17:53] i'm enjoying how the stretch preseed file says the full documentation for partman recipes is in the partman-auto.txt file. [10:18:28] wat. adding $reusemethod() to /srv causes the installer to say No root file system is defined [10:18:30] that's amazing [10:18:56] kormat: see https://jack.einval.com/debian-preseed/ for the full list [10:19:49] kormat: So looks like what I experienced years ago is still there? :( [10:19:59] I think last time we were happy with having a "working" recipe, as syntax changes from version to version and breaks recipes [10:20:02] I don't recall exactly what it was, just that it was a pain [10:20:43] moritzm: that looks like it's "just" the preseed questions, but not documentation on the partman-auto recipe [10:21:02] I wonder if it would be possible to "record" manual actions and then apply them? [10:21:32] i think it's probably time to take a look at the partman-auto source [10:21:37] he [10:21:51] ah, I thought you were looking for a list of all preseed settings [10:21:59] you can find https://github.com/xobs/debian-installer/blob/master/doc/devel/partman-auto-recipe.txt here [10:22:37] moritzm: that one is from 2009, and it was incomplete even then :) [10:22:54] I found this bug, could be relevant? https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/455587 [10:23:10] when your examples section contains parameters that were never mentioned before, you're doing it wrong [10:23:32] jynus: $reusemethod{} was added by cwatson >= 2011 [10:23:38] ok [10:23:38] *cjwatson [10:24:06] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [10:24:16] https://salsa.debian.org/installer-team/debian-installer/-/blob/master/doc/devel/partman-auto-recipe.txt is the most recent one, but I doubt there were many changes since 2009 :-) [10:24:17] partman is amazingly bad. [10:24:24] what? [10:24:27] moritzm: yeah, i've seen that too [10:24:29] how can you say that! [10:24:34] sacrilege [10:24:37] * kormat cowers [10:25:01] jynus: I want to upgrade db2078 to 10.4, it is the misc host used by codfw to extract backups, would you be ok with that? [10:25:06] I think a good GSoC project for next year would be to create a sane utility [10:25:20] to configure and then translate to partman [10:25:40] like the standarization that is happening, but beyond that [10:26:00] marostegui: I think backups finished, let me check [10:26:14] that sounds like a good idea actually [10:26:25] jynus: yeah, won't do it today, but the general question is...are you ok with getting codfw backup source for misc in 10.4? [10:26:44] ah [10:27:01] well, if they are not running at the moment yes, once I discover the issue at the wiki [10:27:12] or you could fix that after reimage [10:27:19] *discovered [10:27:20] yeah, no problem [10:27:37] *at the moment of the reimage [10:27:40] wait! i never puppet-merge'd the reusemethod{} change. so the last run failed _without it_. wat. [10:28:09] kormat: make sure not only to merge, but to make sure it is applied to the right host [10:28:28] it can take 20 minutes for puppet to run on a host [10:28:54] i've been running puppet agent on apt1001 [10:29:03] is that insufficient? [10:29:30] kormat: probably install1003 I think too? [10:29:33] actually, more like 30 [10:29:36] Or that is only for dhcp maybe [10:29:45] yeah, one of the 2, not sure which [10:29:57] the one were it shows the diff :-D [10:30:03] i can see my diff in the run on apt1001 [10:30:08] so that one [10:30:14] yeah, that should be enough [10:30:41] kormat: recently the only host was install1002 but the roles got separated into install hosts and apt hosts a few weeks ago, hence the doubts :) [10:30:49] mentioning because what you did (forgetting to merge) or forgetting to run puppet is a very common issue [10:31:10] we have done somemany times [10:31:17] yeaah, i can imagine :) [10:31:29] specially when in a rush [10:31:33] to deploy [10:31:43] but it's concerning that the partman sometimes throws its toys out of the pram. [10:32:15] so I think that gives context to "yep, that avoids reimage, let's call it a day" [10:32:21] :-D [10:32:26] yep yep yep [10:32:57] but more people available == time to reconsider past decisions/half measures [10:33:05] that's when you enter :-) [10:33:22] same error again. iiinteresting. [10:33:36] time to study this more [10:33:42] use the code, luke [10:38:11] having to go via gerrit+puppet-merge+puppet-agent for each iteration is painful [10:39:38] puppet-merge is a crucial security feature, otherwise exploiting a weakness in Gerrit would result in arbitrary code execution on every server as root [10:40:11] but there's certainly some improvement space for triggered puppet runs as part of the puppet merge [10:40:23] puppet-agent is not normally needed [10:40:27] moritzm: i'm just talking about this specific case [10:40:34] i have no objections in the general scheme of things [10:40:41] but in this case I would be open to allow manual tinkering on server [10:40:48] just for testing [10:40:51] i'm making edits to a file that's only used by one test ganeti instance [10:40:56] asuming you only edit that one file [11:15:56] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10MediaWiki-Special-pages: Optimize recentchanges queries - https://phabricator.wikimedia.org/T251885 (10Marostegui) [12:49:32] hum. it appears that partman-auto-lvm does not allow you to reuse existing lvm vgs/lvs/etc. [12:49:50] so i'll check to see what happens if i tell partman to ignore lvm completely [12:54:30] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10Jclark-ctr) led`s flashing in sequence of 2 most likely a processor error Case Reference ID: 5346998524 Status: Case is generated and in Progress Product: HPE ProLiant DL360 Gen10 8SF... [13:00:10] jynus: I would appreciate a sanity check on https://phabricator.wikimedia.org/P11161 (no rush though, the host is nowhere near to start replicating again) [13:24:36] on it [13:29:18] itwiki/jawiki not bad [13:30:39] yeah, the biggest ones pending are: wikidata, ptwiki and plwiki I think [13:31:21] maybe metawiki too [13:31:36] but at least, we've gone thru dewiki, enwiki, commons, frwiki, es...which are quite big [13:31:52] because it is taking so long [13:32:34] I had thoughts about "importing wb_terms" [13:32:39] it had not more edits, right? [13:32:50] no, in fact it has been dropped, so replication will drop it [13:32:54] except I guess the drop? [13:33:06] marostegui: ok, i'm now reasonably confident there is no way to make partman do anything sane once lvm is in use :( [13:33:09] so maybe we could, not drop the files, but remove all the inserts [13:33:18] kormat: :-( [13:33:28] jynus: don't know... I am not sure I want to experiment with it, it will be the last table pretty much [13:33:31] so maybe not worth the risk [13:33:44] yeah, but it will take a lot to import :-/ [13:34:06] but it will take another week if we have to reimport everything [13:34:12] yeah [13:34:23] what's the binlog retention like? [13:34:28] let me see [13:34:30] is it 1 month? [13:34:33] should be around 30 [13:34:34] yeah [13:34:39] let me check the sanitarium masters [13:34:44] yeah, let's check [13:34:53] while I finish the checks [13:35:05] kormat: so any workarounds? [13:35:11] 15 days on the sanitarium masters [13:35:16] should be fine [13:35:18] jynus: i got nothing [13:35:27] I mean outside of partman [13:35:39] like, literally send keys through serial console [13:36:25] that would be _technically_ possible, but a nightmare [13:36:26] not saying that is the way [13:36:38] I am trying to come up with a strategy [13:36:56] anything that allows us automation [13:37:22] well, one question would be - how strongly do we care about lvm? [13:37:29] he [13:37:40] if we weren't using lvm, i'd be done by now [13:38:19] kormat: we actually don't really take any advtange of it, as we fully expand the vg to the max anyways [13:40:21] the problem is it is there already [13:40:48] so removing it, even if we decide to go in that direction would take at least a round of reimages [13:41:09] yeah :( [13:41:55] the other option is that it can coexist with the other hosts, and as those get reimaged/decommissioned the new ones without lvm will start to show up? [13:41:56] so I would prefer if there was a more practical approach, not saying we shouldn't migrate away, but we need something to help us now [13:42:34] could you explain why the issue happens? [13:42:52] jynus: if d-i detects lvm on the machine, it forces partman-lvm to run [13:42:53] is it partman not having a way to handle lvm? [13:43:05] and that destroys the volumnes? [13:43:07] partman-lvm will not run unless it's allowed to wipe all existing lvm vgs/lvs [13:43:13] I see [13:43:40] :( [13:44:37] even if i tell partman to use 'regular' instead of 'lvm', and tell 'lvmcfg' not to probe lvm vgs, it still happens [13:46:20] I think we should take this to other folks [13:46:31] writing that down on a task [13:46:39] and say, hey, we have this problem [13:47:02] and maybe someone can give us an idea on a way forward [13:47:14] i'll write up my findings in the task i have open for this [13:47:20] thanks [13:53:24] in my defense, I thought it was going to be possible [13:53:43] I just thought I was stupid enough to not being able to solve it on my own [13:53:48] 0:-D [13:54:28] I think we have found our 4th "pain point" for etherpad [13:56:41] other options- I know install images allow to hook extra stuff [13:57:03] moritzm does some magic installing puppet [13:57:19] i've seen a couple of people who have replaced partman with their own horror creations to work around things like this [13:57:25] but.. i'm reluctant do to that :) [13:57:30] maybe we could run a script that would skip the partitioning and do it on our own [13:57:44] well, if it is only to get out of it to a sane (e.g. lvm-less) state [13:58:00] again, not saying "this is it" [13:58:23] just trying to be constructive, throwing ideas [13:58:29] see what sticks [13:59:34] yeah, please write all this down for a wider discussion/analysis with the wider SRE Infra Foundations team, we do have some options for sure, e.g. late-install.sh [13:59:50] late-install.sh too late, but you got our idea :-D [13:59:56] we'll see [14:00:11] that is why I asked for a ticket explaining the whole issue and getting some options [14:00:13] i'm currently digging down into the guts of the partman packages so i can point to line numbers [14:00:27] kormat: are you thinking of a patch? [14:00:31] that is also an option [14:01:36] my concern there is that we'd be changing something which works correctly for everyone else. it will not be easy to have confidence that we're not going to break something [14:01:37] even reporting upstream [14:02:14] who is maintaining partman? [14:03:06] i would guess it's some very unlucky person [14:03:38] no, what I mean is, is this a debian project or a 3rd party that debian just uses [14:03:50] oh it's a debian project [14:04:24] is it? [14:05:03] jynus: yes. and the entire thing is riddled with debconf stuff [14:05:13] yeah, seems to point to https://sources.debian.org/src/partman-auto/ [14:06:07] let's search for a duplicate: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=yes&src=partman-auto [14:06:17] and otherwise report it [14:07:04] even if it is a low priority, feature request, I think there is nothing to lose [14:07:15] sure [14:07:39] *if it is considered as [14:07:57] I think you can take care of that and our task [14:08:19] I will go back to my review for manuel, will give it a thought and share it later [14:11:44] lol @ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=425840 [14:12:23] second most abusing bug I've seen today [14:12:26] *amusing [14:19:53] there is one last alternative, which is "always reimage fully" (destroying /srv), but provision automatically afterwards- the last part is my goal this Q [14:20:34] I don't think manuel would be happy about that :-D [14:20:41] but again, just listing options [14:20:46] or do what we used to...move the data away, fully reimage and then back... but that will take more time and more error prone :( [14:20:51] nah [14:21:04] if we were to do that, we literally have dbprov hosts for that [14:21:18] I just don't think that is ideal, except for partitioning changes [14:21:25] yeah [14:21:36] but data would be forced to be consistent :p [14:21:41] or [14:21:47] be inconsistent everywhere :-D [14:21:55] that's the current situation! :) [14:22:00] Anyways, I am going to go off [14:22:06] have a nice day! [14:22:19] will give you my eval of the paste in a comment [14:22:31] sure, no prob [14:23:27] marostegui: if we were willing to do that _once_ (move /srv data away and back) per host, we could drop lvm, and sleep slightly better [14:24:03] kormat: yeah, that's my point (or at least the hosts we force to reimage entirely, not get lvm back) [14:24:10] Now off for real! :p [14:28:14] kormat: thanks for working on this, this was a much needed work [14:30:56] it's my (dis)pleasure :) [14:51:01] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10jcrespo) Manuel had already done a check before, but double checking again on all servers: ` root@db1120.eqiad.wmnet[(none)]> select table_schema from information_schema.tables where... [15:24:04] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10jcrespo) ` root@cumin1001:~$ table='aft_feedback'; mysql.py -BN -h db1120 -e "select table_schema from information_schema.tables where table_name='$table'" | while read db; do echo "... [15:31:52] 10DBA: Make partman/custom/no-srv-format.cfg work - https://phabricator.wikimedia.org/T251768 (10Kormat) 05Open→03Stalled Blocked by https://phabricator.wikimedia.org/T252027 [15:32:10] @mor [15:32:12] er [15:32:20] moritzm: i filed https://phabricator.wikimedia.org/T252027 with my findings [15:36:33] i don't think there's a simple way to fix this. i'm pretty sure it's fundamental to how partman handles lvm [15:39:07] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10jcrespo) a:05jcrespo→03ArielGlenn CC @ArielGlenn https://dumps.wikimedia.org/other/aft_feedback/20200506/ Only assigning to you for you to ack the above. DBA part (filtering and... [15:44:25] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10ArielGlenn) Let's add that to the 'other' index.html page too, or no one will know it's there. Can someone supply a phrase describing the contents, for downloaders? [15:56:45] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install backup1002 + array - https://phabricator.wikimedia.org/T250816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['backup1002.eqiad.wmnet'] ` The log can be found in `/var/... [16:03:02] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Reedy) >>! In T250715#6113154, @ArielGlenn wrote: > Let's add that to the 'other' index.html page too, or no one will know it's there. Can someone supply a phrase describing the cont... [16:07:20] 10DBA, 10Privacy Engineering, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10jcrespo) > I think the only missing thing might just be a quick sanity check of the dumps Note I did one at T250715#6113043 (the imported results are after reimporting the dumps bac... [16:40:08] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install backup1002 + array - https://phabricator.wikimedia.org/T250816 (10jcrespo) @Cmjohnson I can POST the server and get to BIOS without issue. However, I did the above change^ because the server didn't boot into... [16:40:47] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install backup1002 + array - https://phabricator.wikimedia.org/T250816 (10jcrespo) a:05jcrespo→03Cmjohnson