[05:20:29] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Degraded RAID on es2026 - https://phabricator.wikimedia.org/T263837 (10Marostegui) ` root@es2026:~# megacli -PDRbld -ShowProg -physdrv[32:2] -aALL Rebuild Progress on Device at Enclosure 32, Slot 2 Completed 62% in 875 Minutes. ` [05:29:44] 10Blocked-on-schema-change, 10DBA: Schema change to drop three indexes from wb_changes - https://phabricator.wikimedia.org/T264109 (10Marostegui) [05:38:46] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2016.codfw.wmnet - https://phabricator.wikimedia.org/T264156 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `es2016.codfw.wmnet` - es2016.codfw.wmnet (**PASS**) - Downtimed host on Icinga... [05:39:18] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2016.codfw.wmnet - https://phabricator.wikimedia.org/T264156 (10Marostegui) [05:39:43] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2016.codfw.wmnet - https://phabricator.wikimedia.org/T264156 (10Marostegui) a:05Marostegui→03Papaul Ready for #dc-ops! [05:42:25] 10DBA, 10decommission-hardware: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10Marostegui) [05:42:46] 10DBA, 10decommission-hardware: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10Marostegui) [05:42:48] 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) [05:44:53] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10Marostegui) [05:47:36] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es2011.codfw.wmnet - https://phabricator.wikimedia.org/T264261 (10Marostegui) [07:15:52] 10DBA, 10Operations, 10Sustainability (Incident Followup), 10Wikimedia-Incident: S5 replication issue, affecting watchlist and probably recentchanges - https://phabricator.wikimedia.org/T263842 (10akosiaris) >>! In T263842#6506360, @RhinosF1 wrote: > I just noticed the IR on wikitech says: >>duplicate key... [07:19:09] 10Blocked-on-schema-change, 10DBA: Schema change to drop three indexes from wb_changes - https://phabricator.wikimedia.org/T264109 (10Marostegui) @Ladsgroup this change is deployed to codfw. The master is impossible to alter online, so it will need to get done once we are back in eqiad. I am going to monitor t... [07:42:59] 10DBA: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [07:55:59] 10DBA: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:02:00] 10DBA: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:02:23] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) I think this deserves the #epic task :) [08:26:03] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:27:35] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) Note: Be aware that this bug can bite us when dealing with this task using replication: https://jira.mariadb.org/browse/MDEV-8351 (it just bite me ;-) ) [08:49:21] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:57:31] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Ladsgroup) >>! In T260111#6507725, @Marostegui wrote: > I think this deserves the #epic task :) It will be EPIC :D [08:58:06] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [08:58:52] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) Note: Some of the drifts that are reported only on a host, they also affect hosts that aren't on MW, like dbstore and backup source hosts, so for each host, I am reviewing the whole section, just... [09:18:01] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [10:44:52] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), and 2 others: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10MSantos) 05Open→03Resolved a:03MSantos @Marostegui in case we... [13:38:53] 10Blocked-on-schema-change, 10DBA: Schema change to drop three indexes from wb_changes - https://phabricator.wikimedia.org/T264109 (10Ladsgroup) Thanks! It makes sense. [14:50:01] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10Cmjohnson) @Jclark-ctr I see that the task has been resolved in coupa but I don't see the servers anywhere and they're not in netbox. Where are they? [14:53:39] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Degraded RAID on es2026 - https://phabricator.wikimedia.org/T263837 (10Marostegui) The RAID finished correctly. @Papaul what do you want to do once the new disk arrives? Should we leave this old one in, or should we pull it out and insert the new one? `... [14:56:18] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Degraded RAID on es2026 - https://phabricator.wikimedia.org/T263837 (10Marostegui) 05Open→03Resolved Going to close this as resolved. @Papaul let me know your thought from the above comment! Thank you [19:01:42] 10DBA, 10Operations, 10netops, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10Cmjohnson) 05Open→03Resolved This has been completed [19:25:13] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10wiki_willy) @Cmjohnson - I just asked John and he says these are in shipping. Thanks Willy [20:18:45] 10DBA, 10Operations, 10netops, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10ayounsi) 05Resolved→03Open From the task description: > [DCops] Update Netbox At least the status and name are incorrect (should be asw2-d4 for consistency) > [D... [21:11:43] 10DBA, 10Operations, 10netops, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10wiki_willy) Related to Arzhel's previous comment, getting these Netbox errors: test_missing_assets_from_accounting asw3-d4-eqiad Device with s/n TA3716160376 (WMF542...