[05:48:35] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `es2011.codfw.wmnet` - es2011.codfw.wmnet (**PASS**) - Downtimed host on Icinga... [05:49:59] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10Marostegui) a:05Marostegui→03Papaul [06:03:32] 10DBA, 10decommission-hardware: decommission es2017.codfw.wmnet - https://phabricator.wikimedia.org/T264386 (10Marostegui) [06:04:16] 10DBA, 10decommission-hardware: decommission es2017.codfw.wmnet - https://phabricator.wikimedia.org/T264386 (10Marostegui) This will be started on Monday, we don't gain anything by stopping this host for the weekend. [06:04:43] 10DBA, 10decommission-hardware: decommission es2017.codfw.wmnet - https://phabricator.wikimedia.org/T264386 (10Marostegui) [06:04:46] 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) [06:57:14] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) a:03Marostegui [07:46:09] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [07:49:08] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [07:52:32] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:14:41] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:17:35] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:25:13] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:30:27] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:51:24] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:59:45] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [09:23:18] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [09:45:49] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [10:28:07] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [10:30:20] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [10:33:25] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [10:33:57] Amir1: thank you so much for T260111 you have no idea all the mess s3 is....it was not just only those hosts...I am fixing lots of things along the way. Thanks for filing that task [10:33:58] T260111: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 [10:34:44] marostegui: oooh, what else did you find? I'm curios now [10:35:27] Thank you for cleaning it up. I'm just reporting and nagging [10:36:05] Amir1: There were hosts with different drifts within the same wiki [10:36:19] marostegui: 🙀 [10:36:28] I am still curious on why, for instance ruwikiquote had so many drifts, that particular wiki... [10:36:38] kormat: yeah, it is incredibly messy [10:36:59] I am doing what jaime and myself used to call...archeology work here [10:38:41] https://usercontent.irccloud-cdn.com/file/TjOmyoYF/image.png [10:38:47] Our DBAs [10:38:58] at work [10:38:58] hahaha exactly [10:39:06] with more screaming [10:41:30] lol [10:47:46] ruwikiquote? [10:47:49] * Reedy blames Putin [10:48:17] Reedy: say goodbye to the russian vaccine for you! [10:48:27] Do I need vaccinating against Russian? [10:48:44] haha [10:54:19] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:05:09] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:08:37] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:15:38] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:21:03] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:23:13] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:31:10] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:35:41] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) This one is actually a lack of PK (but at least `user_id` is the UNIQUE) :-( ` "user user_id index-mismatch-prod-extra": [ "db1123:cywikiquote", "db1078:cywikiquote", "db1075:cywikiquote",... [11:40:32] marostegui: Happy Friday [11:40:38] XDDDD [11:44:52] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [11:46:45] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [12:08:36] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [12:32:09] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [13:10:58] es2033 y es2034 están siendo montados, o pendientes de montar quizás? [13:11:10] sorry [13:11:28] was asking about es2033 and es2034, if they are being setup or pending to setup? [13:11:52] they are set up [13:12:02] strange then [13:12:09] what is strange? [13:12:17] "missing physical device in PuppetDB: state Active in Netbox" [13:12:32] I am not sure if that needs manual changing or not? [13:12:32] something may be weird on puppetdb [13:12:59] they show as active on netbox [13:12:59] if they are up active is normal [13:13:20] but something may be missing from puppet db (not necesarilly puppet config) [13:13:33] or maybe just needs refresh or something [13:14:32] I will ask around [13:17:04] Don't worry, I can ask Cas or Riccardo on Monday [13:17:27] I just did already [13:17:41] if I get a response, I will try to fix it [13:17:56] ok [13:18:01] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [13:18:13] I happened to look that becase I wasn't sure I had active'd the recently db source backups hosts [13:18:19] they are not in puppetdb, see https://puppetboard.wikimedia.org/node/es2034.codfw.wmnet [13:18:36] volans|off: how's tha possible? [13:18:47] ok, so how can they be on puppet repo but not on puppet db [13:19:00] is puppet running? [13:19:31] marostegui: "Puppet is disabled. reason not specified" [13:19:44] uh? [13:19:45] volans|off: thanks, mistery solved [13:20:27] Just enabled it, I guess i disabled it for something and forgot [13:20:30] I will let you handle it [13:20:44] yeah, probably just a normal thing that would have bee caught by icinga [13:20:47] so not an issue [13:21:03] feedback request, was the error message in the report not clear enough? "missing physical device in PuppetDB: state Active in Netbox" [13:22:45] I wouldn't change it, I just didn't think that puppet disabled would remove it from puppet db [13:23:18] we have a 1 week auto-GC from puppetdb [13:23:24] cool, thanks [13:23:36] it's up for discussion as there are pro and cons [13:23:40] if you want to go beyond the call of duty, I would just document on wikitech [13:23:57] this as a "common reasons why this alerts" [13:24:06] but I think it was reasonable [13:24:41] ack, thx [13:28:06] "Commonly caused by: marostegui" [13:28:36] I am trying to remember why I disabled puppet on those two [13:28:37] Blame: kormat [13:28:52] (I assume it was me) [14:11:33] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [16:28:27] 10DBA, 10cloud-services-team (Kanban): wikiadmin does not have grants to labtestwiki database - https://phabricator.wikimedia.org/T264456 (10Urbanecm) [17:40:58] 10DBA, 10Operations, 10ops-codfw, 10User-Kormat: db2125 crashed - mgmt iface also not available - https://phabricator.wikimedia.org/T260670 (10wiki_willy) After escalating to technical account rep, replacement CPUs are being shipped by Dell, and can wait to be replaced when Papaul is back from vacation. [19:35:21] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Incomplete i18n for log entries in CheckUser - https://phabricator.wikimedia.org/T41013 (10Niharika) My bad, I just saw this task today. I'll take a look at it. [21:33:54] 10DBA, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-SecurePoll, 10Platform Engineering, 10User-DannyS712: [Beta cluster] Table 'centralauth.securepoll_lists' doesn't exist - https://phabricator.wikimedia.org/T264477 (10DannyS712) [21:41:28] 10DBA, 10Beta-Cluster-Infrastructure, 10Platform Engineering, 10User-DannyS712: [Beta cluster] Table 'centralauth.securepoll_lists' doesn't exist - https://phabricator.wikimedia.org/T264477 (10Reedy) [21:41:51] 10DBA, 10Beta-Cluster-Infrastructure, 10Platform Engineering, 10User-DannyS712: [Beta cluster] Table 'centralauth.securepoll_lists' doesn't exist - https://phabricator.wikimedia.org/T264477 (10Reedy) 05Open→03Resolved a:03Reedy Table created