[05:41:36] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) Thank you! [06:03:01] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) a:03Marostegui [06:15:32] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [06:15:47] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [06:37:04] 10DBA: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Marostegui) a:03Marostegui [06:39:25] 10DBA: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Marostegui) I have renamed the tables, will leave them like this for a few days to make sure nothing breaks ` +----------------------------------+ | Tables_in_designate_pool_manager |... [06:43:37] 10DBA, 10Patch-For-Review: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Marostegui) @Andrew does this need to be removed? https://github.com/wikimedia/puppet/blob/production/hieradata/common/profile/openstack/base/designate.yaml#L7 [06:47:33] 10DBA, 10Operations, 10Patch-For-Review: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Marostegui) I am going to drop the following grants: ` root@db1133.eqiad.wmnet[(none)]> select user,host from mysql.user where user like 'labspuppet'; +-... [07:02:35] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) This is how the table looks like in db2089:3316 after the ALTER: ` frwiki *************************** 1. row *************************** Table: user_newtalk Create T... [07:03:29] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) s6: [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db2129 [] db2124 [] db2117 [] db2114 [] db2097 [] db2095 [x] db2089 [] db2087 [] db2076 [] db... [07:03:42] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [07:09:08] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) This is how the tables look like after this alter and the alter for the special slaves (T233625) ` root@cumin1001:/home/marostegui# for i... [07:21:32] 10DBA, 10Operations, 10Patch-For-Review: Check/remove unused databases following labpuppetmaster deprecation - https://phabricator.wikimedia.org/T233281 (10Marostegui) 05Open→03Resolved database dropped: ` root@db1133.eqiad.wmnet[labspuppet]> show tables; +-------------------------+ | Tables_in_labspuppe... [07:32:12] 10DBA, 10Patch-For-Review: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Marostegui) Apart from the grants at: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540534/1/modules/role/templates/mariadb/grants/production-m5.sql.erb... [07:39:19] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [08:15:14] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [08:23:50] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) ` root@cumin1001:/home/marostegui# for i in db1106 db1074 db1112 db1121 db1082 db1085 db1079 db1087 db2072 db2126 db2074 db2073... [08:24:08] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) [08:24:27] 10DBA: Add "candidate master and sanitarium master" on dbctl for the candidate masters and sanitarium masters in eqiad/codfw - https://phabricator.wikimedia.org/T234039 (10Marostegui) 05Open→03Resolved [08:24:29] 10DBA, 10conftool: #dbctl: add 'comment'/'description' metadata to instances - https://phabricator.wikimedia.org/T229677 (10Marostegui) [12:56:22] 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose New History Queries - https://phabricator.wikimedia.org/T231599 (10Anomie) Good catch. Yes, the `AND (rev_deleted & 4) = 0` check should be left out of the "reverted" query. [13:48:18] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Marostegui) @Cmjohnson es1019 is now off and ready for you. Once you are done, power it back on. Thanks! [14:06:33] 10DBA, 10Patch-For-Review: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Andrew) >>! In T233978#5543342, @Marostegui wrote: > @Andrew does this need to be removed? https://github.com/wikimedia/puppet/blob/production/hieradata/common/p... [14:27:53] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [15:40:24] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Anomie) >>! In T234066#5543364, @Marostegui wrote: > This is how the table looks like in db2089:3316 after the ALTER: All three look correct. Thanks! [15:51:33] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Anomie) >>! In T233135#5543370, @Marostegui wrote: > This is how the tables look like after this alter and the alter for the special slaves (T233625)... [15:52:30] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) >>! In T233135#5544304, @Anomie wrote: >>>! In T233135#5543370, @Marostegui wrote: >> This is how the tables look like after this alter an... [19:41:31] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Cmjohnson) 05Open→03Resolved a:03Cmjohnson I powered the server off, unplugged everything, removed the PSU drained flea power by holding the power button f... [19:46:30] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Marostegui) Thanks - I can access too! [20:46:26] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for nqowiki - https://phabricator.wikimedia.org/T230543 (10Bstorm) [21:53:15] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Dzahn) 05Resolved→03Open ssh to mgmt works now. confirmed. But IPMI from remote fails to establish a session. Could you please check if "IPMI over LAN" is d... [21:55:58] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Dzahn) 05Open→03Resolved I take that comment back. (now) it works for me. [22:01:57] 10DBA, 10Growth-Team, 10PageCuration, 10User-DannyS712: Delete a row from pagetriage_page on beta cluster - https://phabricator.wikimedia.org/T234577 (10DannyS712) [22:03:17] 10DBA, 10Growth-Team, 10PageCuration, 10User-DannyS712: Delete a row from pagetriage_page on beta cluster - https://phabricator.wikimedia.org/T234577 (10DannyS712) 05Open→03Stalled Please wait until after {T234574} is fixed to do this