[05:33:47] 10DBA, 10Data-Persistence, 10decommission-hardware, 10Patch-For-Review: decommission es2015.codfw.wmnet - https://phabricator.wikimedia.org/T264700 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `es2015.codfw.wmnet` - es2015.codfw.wmnet (**PASS**) - Do... [05:35:00] 10DBA, 10Data-Persistence, 10decommission-hardware, 10Patch-For-Review: decommission es2015.codfw.wmnet - https://phabricator.wikimedia.org/T264700 (10Marostegui) a:05Marostegui→03Papaul [05:35:06] 10DBA, 10Data-Persistence, 10decommission-hardware, 10Patch-For-Review: decommission es2015.codfw.wmnet - https://phabricator.wikimedia.org/T264700 (10Marostegui) This host is ready for #dc-ops! [05:36:27] 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) All codfw hosts have handed over to DC-Ops for decommissioning. [05:44:24] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10Marostegui) Thanks for the heads up @ifried - from those Wikipedias, the German and the French ones will be interesting to monitor, as those are quite big alread... [05:47:40] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [05:48:44] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [06:57:24] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [06:57:36] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) 05Open→03Resolved This is all done! [07:14:12] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Ladsgroup) Wohoooo. Thanks! {meme, src=therockapplauds} [07:19:43] ^ haha [09:33:18] 10DBA, 10Data-Persistence: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10LSobanski) [09:34:41] 10Blocked-on-schema-change, 10DBA, 10Data-Persistence, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10LSobanski) [09:45:05] 10DBA, 10Data-Persistence: Remove muswiki and mhwiktionary from s3 - https://phabricator.wikimedia.org/T260112 (10LSobanski) [09:53:41] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10LSobanski) Thanks, let us know when the database is created, so we can sanitize it. [09:55:01] 10DBA, 10Data-Persistence: transfer.py fails when copying data between es hosts - https://phabricator.wikimedia.org/T262388 (10LSobanski) [10:09:16] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10LSobanski) [11:07:29] 10DBA, 10Data-Persistence: Remove muswiki and mhwiktionary from s3 - https://phabricator.wikimedia.org/T260112 (10Urbanecm) >>! In T260112#6382513, @Marostegui wrote: > No errors so far, for the last 2 days. > If all continues like this, on Monday I will remove the tables and the database from s3. Did that ha... [11:08:00] 10DBA, 10Data-Persistence: Remove muswiki and mhwiktionary from s3 - https://phabricator.wikimedia.org/T260112 (10Marostegui) No, I am waiting for eqiad to be back as active [14:21:42] 10DBA, 10Data-Persistence: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) I have noticed some slight increase on InnoDB wait time on pc2009, it could be just a coincidence but I am going to revert back to `innodb_change_buffering =...