[04:14:33] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Legoktm) >>! In T202904#4536081, @Anomie wrote: > The task itself appears to be non-public. I'll leave it to others to decide whether to change that or provide details, bu... [05:05:09] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1121 [] db1103 [] db1102 [] db1097 [] db1091 [] db1084 [] db1081 [] db1068 [05:05:20] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1121 [] db1103 [] db1102 [... [05:05:29] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1121 [] db1103 [] db1102 [] db1097 [] db1091 []... [05:05:45] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [05:06:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [05:06:13] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [06:55:07] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10jcrespo) Is the documentation of creating a wiki clear enough? Should it be added to other dblists, aside from the section one? [06:56:19] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) >>! In T202904#4536081, @Anomie wrote: >>>! In T202904#4535114, @Marostegui wrote: >> I will check all the schema changes we have done since 18th April then.... [06:57:34] ^I think it would be easier to just compare to another s3 wiki and check for differences [06:58:28] that is what I said there :-) (or what I wanted to say) [06:58:35] check with a recent wiki [07:00:41] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T202824 (10Marostegui) 05Open>03Resolved All good! Thank you! ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK) physicaldrive 1I:1:2 (port... [07:00:56] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10jcrespo) >>! In T202904#4537125, @jcrespo wrote: > Is the documentation of creating a wiki clear enough? Should it be added to other dblists, aside from the section one?... [07:18:39] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Marostegui) I have started to compare the main tables for s1 and s2 as the server caught up already [07:37:24] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) Missing schema changes (I have compared it with the wiki created at T201001 T192926 T196379 T195193 T197891 T199368 [07:37:52] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) [07:38:11] 10Blocked-on-schema-change, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Nikerabbit) [07:38:25] 10Blocked-on-schema-change, 10MediaWiki-extensions-Translate, 10Language-2018-July-September: Apply schema change to translate_reviews in WMF - https://phabricator.wikimedia.org/T201011 (10Nikerabbit) [07:43:35] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [07:43:45] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [07:43:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [07:59:27] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) [10:51:49] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [10:53:45] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [10:57:25] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [11:01:57] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) Grants added on m5 for the backups: ``` | GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `nova_api_eqiad1`.* TO 'dump'@'10.192.0.25'... [11:02:28] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [11:05:52] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) 05Open>03Resolved All is done. Thanks @Bstorm, @Marostegui and @jcrespo !! [11:06:45] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) The backups will run today, so we will check tomorrow if they were generated correctly [12:05:30] marostegui: would it be easy for you to grab the create index statement for the tmp1 index on the wikidata wb_terms table for me? [12:05:40] addshore: yep [12:05:44] great! :D [12:05:45] one sec [12:09:22] addshore: CREATE INDEX `tmp1` ON wb_terms (`term_language`,`term_type`,`term_entity_type`,`term_search_key`); [12:10:10] I don't know why, but I was expecting something more complex, heh, time for a cup of tea [12:11:10] :) [12:13:34] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Marostegui) s2 finished checking - all good [13:53:24] 10DBA, 10Core-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 4 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044 (10Anomie) [14:43:13] db2042 (m3) BBU is recharging [14:43:18] we'll see if it goes fine [14:43:40] is that new? [14:43:49] new as in new server? [14:43:57] the alert [14:44:03] ah, it is a warning [14:44:04] yes [14:44:05] it is new [14:44:11] didn't see that [14:44:17] let me see if it happened before [14:44:17] It is recharging maybe the learning cycle [14:44:26] yeah, I was looking at taht [14:44:34] we have apparently some host without it disabled [14:44:43] probably bios / he fix [14:44:48] *hw [14:44:52] or we forgot [14:44:56] e.g. labsdb1005/4 [14:45:05] yeah could be [14:45:08] they are raid6 and have some problematic disk [14:45:14] unrelated to your mention [14:45:23] just from that I started checking some RAID configs [14:45:40] https://phabricator.wikimedia.org/T202051 [14:45:52] happened on the 16 [14:45:58] Ah, that's when I was out [14:45:58] so we may have an ongoing issue [14:46:06] should I reopen? [14:46:11] or we try once more [14:46:16] let's see if it finishes correctly [14:46:21] (it started by itself) [14:46:24] can you at least note it? [14:46:27] just a comment [14:46:28] yeo [14:46:45] so if it happens again, it is a 3rd time [14:47:49] 10DBA, 10Operations, 10ops-codfw: db2042 RAID battery failed - https://phabricator.wikimedia.org/T202051 (10Marostegui) This server started to recharge its BBU again: ``` WARNING: Slot 0: OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Controller: OK -... [14:56:43] I just saw it is m3, so not a huge priority right now [14:57:03] yeah, I said it earlier, m3 [14:57:13] I will start thinking about warming up s* hosts [14:57:35] as errors may only be revealed under stress [14:57:38] yeah, and es and pc [14:57:41] I will sync with you [14:57:49] koi [14:57:50] oki [14:57:57] so I don't do it while schema changes are ongoing [14:58:36] yeah, check the agenda we did: https://wikitech.wikimedia.org/wiki/Switch_Datacenter/planned_db_maintenance#2018_Switch_Datacenter [14:59:41] I was hoping to delay maintenance if possible and do warmup test in advance [14:59:59] sure, I am doing 1 section at the time only for the schema changes [15:00:07] so we can test with the other 7 sections [15:00:10] right now I am doing s4 [15:00:13] yes [15:00:46] I need to at least finish s4 (not the master) so I can alter that s4 master while eqiad isn't active as the externallinks table is 300G and I prefer to alter it while it is not active [15:01:37] +1 [15:02:12] I have a plan to reduce size of *links tables, but I don't want to push hard for it without comment, MCR, actor changes first [15:02:25] btw, are comment changes part of the scheduled ones? [15:02:38] yeah, the masters [15:03:24] but only the first phase [15:03:54] we should also not forget about wikitech [15:04:06] to alter those? [15:04:12] yeah, it is included on the tasks now [15:04:12] all, in general [15:04:15] yeah [15:04:21] It is now being included as default [16:35:32] 10DBA, 10Operations, 10decommission, 10Goal: reclaim and return all cisco servers - https://phabricator.wikimedia.org/T128821 (10Cmjohnson) [16:35:36] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807 (10Cmjohnson) [16:42:58] 10DBA, 10Operations, 10decommission, 10Goal: reclaim and return all cisco servers - https://phabricator.wikimedia.org/T128821 (10Cmjohnson) a:03RobH All CISCO servers are off the rack and fully decommissioned. I do not have any clue on what to do next with them but would love to get them out of the stora...