[05:25:14] 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) [05:25:28] 10DBA, 10Goal: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10Marostegui) [06:28:13] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2070 - https://phabricator.wikimedia.org/T219852 (10Marostegui) p:05Triage→03Normal a:03Papaul Can we get this disk replaced? Thanks! [06:28:45] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [06:28:55] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [06:35:11] 10Blocked-on-schema-change, 10Notifications, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Marostegui) a:05Catrope→03Marostegui I am going to do a test with these schema changes, as the... [06:51:05] 10Blocked-on-schema-change, 10Growth-Team, 10Notifications, 10Patch-For-Review, 10Schema-change: Add index on event_page_id on echo_event table - https://phabricator.wikimedia.org/T143961 (10Marostegui) 05Open→03Resolved All done [08:05:31] 10DBA, 10MediaWiki-extensions-UrlShortener, 10Patch-For-Review: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10jcrespo) It was empty or almost empty (we should double check on all hosts), a single command on the master should be enough to compress, Marostegui you do it or I do it? [08:05:46] ^marostegui I can take care [08:06:00] jynus: up to you! I am fine doing it to :) [08:06:27] I don't know, it won't take long [08:06:40] you do it then! :-) [08:06:44] +1 [08:07:04] 10DBA, 10MediaWiki-extensions-UrlShortener, 10Patch-For-Review: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10jcrespo) a:03jcrespo [08:07:09] And once you are done, can you check: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/500654/ (no rush) [08:07:53] are you planning on executing it on the master? [08:08:05] downtiming all other hosts? [08:08:13] or just running it on the replicas? [08:08:24] no, on the master (downtiming the hosts, yep) [08:08:25] I guess the first because row [08:08:31] exactly, that is the poiint [08:08:33] honestly, if you are going to do that [08:08:55] I prefer you to do T219777 at the same time [08:08:55] T219777: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 [08:09:02] sure! [08:09:23] problem is you may find problems with too many reads/metadata locking on certain tables [08:09:51] check those are not accessed as frequently as,eg recentchanges, revision, page [08:09:58] and use low timeouts [08:10:15] also make sure the replicas get depooled from the jobqueue [08:10:18] I am going to use only a few wikis and small ones [08:10:21] just to test [08:10:30] not going for the big ones yet [08:10:39] otherwise it is ok, do you know which time is the best for doing that? [08:10:56] db1069 may take some time [08:10:58] so the point is to see how serving reads from the master goes, and then decide [08:11:06] make sure to warmup the tables first (SELECT) [08:11:17] oh, that doesn't worry me [08:11:23] I am thinking beyond that [08:11:32] yeah but I am not going to go for the whole thing yet [08:12:01] so, I do the url compression [08:12:13] You ok with the patch then? [08:12:51] 10DBA, 10MediaWiki-extensions-UrlShortener, 10Patch-For-Review: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10Marostegui) a:05jcrespo→03Marostegui As per our last IRC chat, I will compress it :) [08:12:58] maybe add a comment with the ticket [08:13:08] if it is going to be depooled for a long time [08:13:16] no, it will be like 5-10 mins tops [08:13:23] ok [08:13:49] you are too careful with x1 [08:13:56] it used to be just one host :-) [08:13:59] hahaha [08:14:26] that is the other side of the spectrum! [08:14:48] make sure the usual suspects are enabled on compression [08:14:53] hehe yeah [08:14:55] barracuda, etc. [08:16:58] I will see what I can do with backups [08:27:20] 10DBA, 10Wikidata, 10Wikidata wb_terms Trailblazing, 10User-Addshore: Send the schema to DBAs and follow up with their review - https://phabricator.wikimedia.org/T219142 (10alaa_wmde) a:05Addshore→03alaa_wmde [08:27:34] 10DBA, 10Wikidata, 10Wikidata wb_terms Trailblazing, 10User-Addshore: Send the schema to DBAs and follow up with their review - https://phabricator.wikimedia.org/T219142 (10alaa_wmde) https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/499142#message-4a5c6dfa27a45712c0c22de663943d5797df0e7d [08:38:22] 10DBA, 10MediaWiki-extensions-UrlShortener, 10Patch-For-Review, 10User-Ladsgroup: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10Marostegui) a:05Marostegui→03Ladsgroup This table has been compressed everywhere: ` root@cumin1001:/home/marostegui# ./section x1 | while read hos... [08:38:45] marostegui: jynus so the extension is good to go in your opinion? [08:39:23] (After merging the gerrit change ofc) [08:39:40] Amir1: Once merged the limit, can it be enabled gradually? or is it ON/OFF? [08:39:48] yeah, once marostegui finishees the maintenance on x1 [08:39:55] ^ yep [08:40:31] I think it can be enabled without a lot of promotion first [08:40:35] marostegui: it's ON/OFF basically but we are not enabling tools that might use it [08:40:53] then do an official announcement/using it on tools as amir says one by one [08:41:10] yup [08:59:38] Amir1: I am done with x1 [08:59:54] Thanks. I wouldn't be deploying it until tomorrow [09:00:28] great [09:00:36] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) Thanks, it installed with no issues. [09:00:40] let us know so we can also keep an eye [09:03:27] 10Blocked-on-schema-change, 10Notifications, 10MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Marostegui) So, I have run both changes directly on the master with replication enabled on the fol... [09:37:52] 10DBA, 10MediaWiki-Configuration, 10Operations, 10Patch-For-Review, and 2 others: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126 (10Joe) [09:54:48] I am going to take a break, please have a look at the ideas at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500683/ [09:54:57] cool [09:54:58] enjoy [09:55:35] we should either deploy that, or something different today, probably [09:56:10] yeah, agreed [10:05:53] 10DBA, 10MediaWiki-extensions-UrlShortener, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), 10Patch-For-Review, 10User-Ladsgroup: DBA review of UrlShortener - https://phabricator.wikimedia.org/T219777 (10Ladsgroup) 05Open→03Resolved \o/ [12:13:29] marostegui: thank you to have filled up a task for contint1001 /srv disk filling up ;] [12:13:34] or I would surely have missed the alarm [12:13:46] hashar: you are welcome! [15:07:37] disabled cron manually from es2001 [15:07:45] ok [15:08:23] I am not 100% sure of this being the right final setup, but better have something working and later change it [15:08:27] that doing nothing [15:08:37] yeah, better to know what we are facing :) [15:09:09] puppet run without issues everywhere and with expected outcome [15:09:44] \o/ [15:28:28] hey .. just quick question for now.. @marostegui @jynus I tagged you T219145 .. was wondering if I should add any tags to the phab ticket too for your convenience? [15:28:28] T219145: Pass migration plan to DBAs for review - https://phabricator.wikimedia.org/T219145 [15:29:50] alaa_wmde: we are both subscribed, that is ok I think [15:29:56] I just asked you a question there actually :) [15:44:54] 10DBA: Decommission 2 codfw x1 hosts db2033 and db2034 - https://phabricator.wikimedia.org/T219493 (10Marostegui) I have finished running a compare.py for `wikishared` between db2033 and db2069 and they are all the same. Also ran a compare.py for the following tables across all the databases: ` echo_event echo_... [15:45:05] 10DBA, 10CirrusSearch, 10Discovery, 10Discovery-Search, 10Schema-change: Change job table to use mediumblob for job_params field - https://phabricator.wikimedia.org/T219887 (10EBernhardson) [15:46:07] 10DBA, 10CirrusSearch, 10Discovery, 10Discovery-Search (Current work), 10Schema-change: Change job table to use mediumblob for job_params field - https://phabricator.wikimedia.org/T219887 (10EBernhardson) a:05EBernhardson→03None [15:57:01] 10DBA, 10CirrusSearch, 10Discovery, 10Discovery-Search (Current work), 10Schema-change: Change job table to use mediumblob for job_params field - https://phabricator.wikimedia.org/T219887 (10Marostegui) p:05Triage→03Normal Confirmed the table is empty in core. So the schema change should be able to b... [16:04:39] marostegui: sorry just saw this msg .. I replied but need @Amir1 to confirm [16:05:08] alaa_wmde: I just answered :P [16:09:24] marostegui: jynus It would be great if you have some time to comment on https://phabricator.wikimedia.org/T219805 if you think we should not use doctrine [16:09:41] Amir1: we will [16:13:03] subscribed you both to it for your convenience :) [17:03:24] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Mholloway) Does `wikimedia_editor_tasks_entity_description_exists` being moved to the l... [17:22:00] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Marostegui) >>! In T218302#5078358, @Mholloway wrote: > Does `wikimedia_editor_tasks_en... [17:30:04] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Mholloway) There's nothing private on that table. It only contains data on whether a d... [17:34:17] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Marostegui) Then yeah, I guess we have to remove it from the filters, copy it to sanita... [17:39:11] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Reedy) That table looks fine to be made public [17:42:42] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Marostegui) @Mholloway the one on x1 can be deleted then? ` root@db1069:/srv/sqldata# m... [17:43:39] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Mholloway) @Marostegui Yes, that's fine to delete. Thank you. [18:11:55] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Tgr) There's nothing private in the table, but probably nothing worth taking up resourc... [18:34:05] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Marostegui) Having a table under `$private_tables` means it will not be replicated (rep... [21:34:20] 10DBA, 10Jade, 10Scoring-platform-team (Research): Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10Halfak)