[06:42:01] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) >>! In T266483#6605480, @Kormat wrote: > es1 eqiad: > [x] es1012 > [x] es1016 > [] es1018 > [x] es1027 > [x] es1029 >>! In T266483#6605481, @Kormat wrote: >... [06:42:25] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [07:06:25] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) On going transfers: db1124:s1 -> clouddb1013 db1124:s3 -> clouddb1013 Coordinates: {P13279} [07:06:40] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [07:54:17] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [07:54:38] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) 05Open→03Resolved Almost... [07:57:21] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) a:03Marostegui [07:57:34] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) a:03Marostegui [07:58:50] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [07:59:09] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:00:50] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) @Ladsgroup do you have the gerrit URL for this patch? The above gives 404. [08:25:50] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [08:28:34] Something is going on with tendril - I am investigating [08:29:26] probably doesn't like all the recent attempts at replacing it :-P [08:29:51] Ah, it is the clean up query, that is taking longer than expected [08:42:31] Should be back now [08:42:40] I truncated the monster table global_status_log [08:49:11] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) [08:50:04] 10DBA, 10Patch-For-Review: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Marostegui) The following hosts have been fully pooled in production: es1033 es2 es1034 es3 [09:09:25] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Ammarpad) [09:10:22] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Ammarpad) >>! In T267335#6626570, @Marostegui wrote: > @Ladsgroup do you have the gerrit URL for this patch? The above gives 404. Updated to gerrit link. [09:11:26] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) Thank you! [09:18:44] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) s6 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db2141 [] db2129 [] db2124 [] db2117 [] db2114 [] db2097 [] db2095 [] db2089 [] db2087... [09:18:46] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) s6 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db2141 [] db2129 [] db2124 [] db2117 [] db2114 [] db2097 [] db2095 [] db2089 [] db2087 [... [09:23:45] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) I have deployed this change to db2089:3316 and I will leave it a couple of days to make sure nothing breaks, replication-wise. ` # for i in `cat /home/marostegui/git/mediaw... [09:24:37] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) I have deployed this change to db2089:3316 and I will leave it a couple of days to make sure nothing breaks, replication-wise. ` # for i in `cat /home/marostegui/git/mediawik... [09:27:06] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [09:27:20] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [09:32:27] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [09:36:51] 10Blocked-on-schema-change: Schema change for renaming namespace_title index on watchlist - https://phabricator.wikimedia.org/T268004 (10Ladsgroup) [09:39:02] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [09:46:20] 10Blocked-on-schema-change: Schema change for renaming namespace_title index on watchlist - https://phabricator.wikimedia.org/T268004 (10Marostegui) Renaming an index isn't supported on mariadb 10.1 or 10.4. It has been added to 10.5 (https://mariadb.com/kb/en/mariadb-1052-release-notes/) but we are not there ye... [09:48:29] 10DBA, 10Orchestrator: Investigate hostname/fqdn handling in orchestrator - https://phabricator.wikimedia.org/T267929 (10Kormat) [09:51:29] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [10:04:15] 10DBA, 10Orchestrator: Investigate hostname/fqdn handling in orchestrator - https://phabricator.wikimedia.org/T267929 (10Kormat) 05Open→03Resolved Conclusion - these settings are required: - `"HostnameResolveMethod": "cname"` - causes bare hostnames to be resolved to fqdn's correctly. - `"MysqlHostnameReso... [10:04:17] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Kormat) [10:15:45] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [10:16:18] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [10:17:26] 10Blocked-on-schema-change: Schema change for renaming namespace_title index on watchlist - https://phabricator.wikimedia.org/T268004 (10Ladsgroup) >>! In T268004#6626801, @Marostegui wrote: > Renaming an index isn't supported on mariadb 10.1 or 10.4. It has been added to 10.5 (https://mariadb.com/kb/en/mariadb-... [10:17:40] 10Blocked-on-schema-change: Schema change for renaming namespace_title index on watchlist - https://phabricator.wikimedia.org/T268004 (10Ladsgroup) [10:21:48] 10Blocked-on-schema-change, 10DBA, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Kormat) [10:22:38] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [10:22:57] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) pc2 eqiad done: ` root@pc1008:~# mysql -e "select @@report_host" +--------------------+ | @@report_host | +--------------------+ | pc1008.eqiad.wmnet | +-... [13:31:45] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6626544, @Maroste... [15:12:04] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [15:16:41] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [15:18:56] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [15:20:13] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [15:38:43] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [15:40:23] 10DBA, 10Data-Services, 10User-Kormat, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10Marostegui) [15:48:17] 10DBA, 10Data-Services, 10User-Kormat, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10jcrespo) While some package changes could be needed, one thing to note is some work could be done with just puppet- When we cr... [16:17:52] there is some leftover files related to T260511 / db1141 on backup1002 [16:17:52] T260511: Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 [16:18:19] wrong ticket [16:18:24] related to T249188 [16:18:25] T249188: Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 [16:18:33] Do you want me to nuke them? [16:18:49] is it ok? [16:19:05] I can take care, just want to make sure they are not needed [16:19:10] yep, not needed [16:19:19] can you verify the dates? [16:19:34] May 27 to Jun 2 [16:19:40] is the last modification [16:19:40] yeah, kill them [16:20:39] not very important, but I was researching why backups on eqiad and codfw had different sizes [16:20:49] and I think it was just that [16:21:18] (making sure we weren't missing data on one of the dcs) [16:33:37] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10RobH) >>! In T261405#6624984, @wiki_willy wrote: > @Jclark-ctr - can you double-check the S/N for db1139. We're getting the following Netbox error: >... [16:50:50] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10RobH) @Jclark-ctr is taking this over, as the mainboard swap did not fix the memory and CPU errors. phab won't let me upload the IML file, so emailing... [17:57:06] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Jclark-ctr) @jcrespo will need downtime for host to remap dimms per HPE [18:06:45] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10RobH) >>! In T261405#6627790, @RobH wrote: >>>! In T261405#6624984, @wiki_willy wrote: >> @Jclark-ctr - can you double-check the S/N for db1139. We're... [18:10:07] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo) @Jclark-ctr I just stopped the host and downtimed it for almost a day, thank you! [19:11:19] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo) {F33917836} {F33917835} [22:56:53] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Jclark-ctr) @jcrespo. replacement dimms should arrive Thursday. Unsure what time they will arrive we can shoot for Thursday. If they arrive late it...