[06:33:02] 10Blocked-on-schema-change: Schema change for renaming two indexes of site_identifiers - https://phabricator.wikimedia.org/T273361 (10Marostegui) [06:33:18] 10Blocked-on-schema-change: Schema change for renaming two indexes of site_identifiers - https://phabricator.wikimedia.org/T273361 (10Marostegui) 05Open→03Resolved All done [06:59:10] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) a:03Marostegui [07:05:36] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) [07:06:16] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) p:05Triage→03Medium [07:27:28] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) s6 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db2141 [] db2129 [] db2124 [] db2117 [] db2114 [] db2097 [] db2095 [] d... [07:28:28] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) Altered one host in codfw, leaving it for the weekend before going for one in eqiad. [07:28:42] 10Blocked-on-schema-change: Schema change for dropping defaults of ipb_timestamp and ipb_expiry - https://phabricator.wikimedia.org/T273358 (10Marostegui) [08:00:09] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Technical-Debt, and 2 others: Make wb_changes_dispatch.chd_seen unsigned in production - https://phabricator.wikimedia.org/T273874 (10Marostegui) a:03Marostegui [08:00:34] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Technical-Debt, and 2 others: Make wb_changes_dispatch.chd_seen unsigned in production - https://phabricator.wikimedia.org/T273874 (10Marostegui) Wikis: ` #cat dblists/wikibaserepo.dblist | grep -v "#" commonswiki testcommonswiki testwikidatawiki wikidataw... [08:01:42] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Technical-Debt, and 2 others: Make wb_changes_dispatch.chd_seen unsigned in production - https://phabricator.wikimedia.org/T273874 (10Marostegui) [08:14:01] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Technical-Debt, and 2 others: Make wb_changes_dispatch.chd_seen unsigned in production - https://phabricator.wikimedia.org/T273874 (10Marostegui) I have altered `testwikidatawiki` (the connection errors are hosts that are down for maintenance and will get th... [08:14:21] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Technical-Debt, and 2 others: Make wb_changes_dispatch.chd_seen unsigned in production - https://phabricator.wikimedia.org/T273874 (10Marostegui) [08:16:15] funnily, after rebuilding logicaly s2 and s3 backups, we got a 7-12% reduction in size [08:29:35] mmmm [08:29:45] ah, logically [08:29:47] makes sense [09:12:58] 10DBA, 10decommission-hardware: decommission db1093.eqiad.wmnet - https://phabricator.wikimedia.org/T273955 (10Marostegui) [09:14:00] 10DBA, 10decommission-hardware: decommission db1093.eqiad.wmnet - https://phabricator.wikimedia.org/T273955 (10Marostegui) [09:14:03] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [09:14:28] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [09:18:42] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10jcrespo) [09:22:56] marostegui: fyi for https://phabricator.wikimedia.org/T273874 the only site that really uses this table is "wikidatawiki", its empty on commons for example :) [09:24:39] addshore: thanks! I checked the test* ones only. I will get to the rest next week as I don't want to do more on a friday :) [09:42:11] <3 [10:20:18] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1157 is now replicating, I will start pooling it on Monday. [11:59:56] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install db11[76-84] - https://phabricator.wikimedia.org/T273566 (10Marostegui) [12:05:25] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install db11[76-84] - https://phabricator.wikimedia.org/T273566 (10Marostegui) These hosts have been added to puppet with: `insetup` role and also assigned a `partman` recipe for the installation. The only puppet change ne... [12:43:00] 10DBA, 10Orchestrator: Cleanup heartbeat.heartbeat on all production instances - https://phabricator.wikimedia.org/T268336 (10Marostegui) x1 cleaned [12:43:11] 10DBA, 10Orchestrator: Cleanup heartbeat.heartbeat on all production instances - https://phabricator.wikimedia.org/T268336 (10Marostegui) [15:48:09] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Cmjohnson) [15:48:58] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Cmjohnson) [15:50:09] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Marostegui) @Cmjohnson this looks goods to me. If possible, can you provide the new IP the host will have before the move so I can set it on the host before powering it off? Thanks! [15:50:21] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Marostegui) p:05Triage→03Medium [15:58:06] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Cmjohnson) @marostegui: the same IP, I am not changing the vlan [15:59:11] 10DBA, 10ops-eqiad: eqiad: move db1111 to rack A8 - https://phabricator.wikimedia.org/T273982 (10Marostegui) Excellent! I will have the host ready for you.