[10:24:01] I am going to move db1077 to replicate from pc1, instead of es1 [10:24:03] *s1 [10:24:14] ok [10:24:47] feel free to rename it test-pc1 if that would be useful [10:24:59] yep, I will do that [10:25:16] marostegui: note you'll have to update `modules/profile/types/mariadb/valid_section.pp` [10:25:40] kormat: ah thanks, I wasn't aware! [10:25:52] i wish there was a way to just match `test-*` in a puppet enum [10:26:37] hmm, actually there might be [10:27:13] jbond42: currently `type Profile::Mariadb::Valid_section` is an Enum. would it be possible to tell puppet that it should either be an enum, or a string matching a given pattern? [10:30:19] something like `Variant[Pattern[/^test-.*/], Enum[ ... current definnition ..]` [11:02:59] kormat: beware of becoming a puppet hacker. Some folks (me) might consult with you [11:03:08] * kormat hides [11:48:05] kormat: yes thats should be completly possible [11:48:53] jbond42: if i did that, would it still be possible to loop over the enum in a template? [11:49:01] or would i have to declare the enum as a separate type [11:50:04] oh yes i forgot that you did that, in that case i would maybe putthe enums in a type of there own e,g, Profile::Mariadb::Valid_section::Enum. then have Profile::Mariadb::Valid_section = Variant[Profile::Mariadb::Valid_section::Enum, Pattern[/test-.*/]] [11:50:42] ah hah. ok, perfect. thanks :) [11:51:02] np [13:16:30] we start early with the image issues- there is 37 images lacking sha1 on the oldimage table [13:16:58] I would expect the oldimage table to be a complete mess :( [13:19:45] In practice, does that mean that if we try to restore an image we cannot be sure if we're restoring the right one? [13:20:00] Or that we cannot find the image to restore at all [13:20:16] no, much worse [13:20:17] And I mean the post-delete restore, not a backup restore [13:20:43] there are rows that may or may not invalid data [13:21:06] for example, there are rows called '' (length 0 string) [13:21:28] and it is a table with no primary key [13:21:46] so the same image may be there even multiple times [13:22:37] I was sure I was going to find inconsistencies- I just didn't know it was this early (just trying to list existing images) [13:24:35] (this impacts beyond backups, impacts undelying application) [13:25:59] I recall there was a task for fixing oldimage [13:26:15] Ah yes, from 2011: https://phabricator.wikimedia.org/T28741 [13:26:17] yes, but on rfc, everybody agreed it had to be done [13:26:24] but no resources given [13:26:41] And MCR has probably changed things anyways [13:27:12] not necesarilly, they are quite independent [13:27:18] probably just the actor stuff [16:16:49] kormat: I will add dborch to https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers [16:19:33] ^that could be a useful url for sobanski too [16:20:06] Bookmarked. Thanks! [16:29:07] I've claimed some ownership there on a best guess, but I may have made some mistakes [17:48:18] Nice [17:48:36] Sorry, wrong channel. [19:01:45] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` es1026.eqiad.wmnet ` The log can be found in `/v... [19:14:41] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1026.eqiad.wmnet'] ` Of which those **FAILED**: ` ['es1026.eqiad.wmnet'] ` [19:50:37] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` es1026.eqiad.wmnet ` The log can be found in `/v... [20:15:07] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1026.eqiad.wmnet'] ` and were **ALL** successful. [20:16:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10RobH) [20:17:35] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` ['es1027.eqiad.wmnet', 'es1028.eqiad.wmnet', 'es... [20:42:31] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1027.eqiad.wmnet', 'es1029.eqiad.wmnet', 'es1028.eqiad.wmnet', 'es1033.eqiad.wmnet', 'es1031... [20:49:13] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10RobH) [20:51:02] 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10RobH) [20:51:11] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10RobH) a:05RobH→03None [20:51:16] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10RobH) 05Open→03Resolved All installations complete and hosts are calling into puppet. They've all been set to staged in netbox, and the #DBA team... [20:56:53] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10wiki_willy) a:03Cmjohnson Thanks @Cmjohnson and @RobH for prioritizing this one. Nice work getting it turned over in time. Thanks, Willy